dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: dohertj2/ScadaBridge:c6769ee95e35feba12abda617124909bf02a408b
Branches Tags
dohertj2/ScadaBridge:main
...
compare: dohertj2/ScadaBridge:15e5887b52ebb5dabfb073cb11dec1ea9eec4408
Branches Tags
dohertj2/ScadaBridge:main

68 Commits
c6769ee95e ... 15e5887b52

Author SHA1 Message Date
Joseph Doherty 15e5887b52 docs(register): row-85 live-probe evidence (Created DCL connection fired alongside the sweep self-heal in the original failure sequence — no second restart, all tags Good); row-96 evidence from the close-out run (Playwright drill-in nav timeout under parallel load, green isolated) 2026-08-17 04:33:33 -04:00
Joseph Doherty 2b1e220f7c docs(register): close row 82 (cross-process create+migrate semaphore on sp_getapplock + honest timeouts + deterministic actor stops; premise corrected — per-run DB names already existed); add rows 95-96 (systemic leaked-tick-actor pattern in three more classes, uncapped xunit parallelism policy) 2026-08-17 04:10:14 -04:00
Joseph Doherty 75195fc512 merge(agent-c): row 82 — cross-process create+migrate throttle (sp_getapplock counting semaphore), honest command timeouts, deterministic actor stops in the two leaking AuditLog test classes 2026-08-17 04:09:36 -04:00
Joseph Doherty da23f2dcae test(mssql): close register row 82 — throttle cross-process create+migrate, stop leaked test actors 
AuditLog.Tests flaked under parallel full-solution runs. Three distinct
mechanisms, all load-sensitive and all green in isolation.

1. MsSqlMigrationFixture create+migrate saturation (the register's headline).
   Database NAMING was already isolated (a guid per fixture), but nothing
   bounded how many fixtures ran CREATE DATABASE + the ~100-migration chain at
   once. xunit parallelises processor-count collections PER PROCESS, and a
   solution-level `dotnet test` starts several MSSQL-backed processes together,
   so the dev container saw dozens of simultaneous fresh-database migrations.
   That is resource saturation, not object-level locking, and it pushed
   individual DDL statements past EF's default 30s command timeout — every
   failure had the identical fingerprint the register recorded:
   MsSqlMigrationFixture..ctor -> Migrator.MigrateAsync -> SqlException
   "Execution Timeout Expired" (error 258), which xunit surfaces as
   "Class fixture type ... threw in its constructor" against every test in the
   class.

   Fix: a machine-wide counting semaphore over the create+migrate window, built
   from sp_getapplock on master (ScadaBridgeTestMigrate/0..N-1, default 4 slots,
   SCADABRIDGE_MSSQL_TEST_MIGRATE_SLOTS to override). An xunit collection cannot
   do this — the contention is cross-PROCESS, so the arbiter has to live on the
   shared server. The gate connection is unpooled (a pooled connection defers the
   implicit release of a session-scoped app lock until reuse) and releases
   explicitly; a crashed process drops its connection and SQL releases the lock,
   so the gate cannot wedge the suite, and a deadline miss proceeds ungated
   rather than failing. Migration commands also get a 300s timeout and
   CREATE/DROP DATABASE 120s: under a bounded queue the server is busy but
   progressing, so waiting is the correct response. Plus a once-per-process
   best-effort sweep of ScadaBridgeAuditMigTest_* databases older than 2h so a
   killed run does not leave strays, and the connectivity probe now retries
   (3 attempts) — a busy server misread as an absent one would silently turn
   every MSSQL test into a Skip, which reads green and is worse than a red.

2. PartitionPurgeTests leaked AuditLogPurgeActors. Each ticks every 100ms and
   switches out the Jan-2026 partition; they were only stopped implicitly by
   TestKit Dispose -> Shutdown, which is timeout-bounded and does not fail when
   it expires. Under load a previous test's actor was still ticking against the
   shared per-class fixture DB, so PerChannelOverride_... found its seeded
   Jan-15-2026 row already gone (Expected: 1, Actual: 0).

3. NotifyDispatcherAuditTrailTests leaked NotificationOutboxActors, same shape.
   The dispatcher claims EVERY due row, not just the test's siteId, so a leaked
   actor delivered the next test's notification through its own exhausted
   QueuedOutcomeAdapter — a transient outcome, leaving the row in Retrying
   (Expected: Delivered, Actual: Retrying) and audit counts short.

   Both fixed by stopping the actor inside the test that created it
   (Watch/Stop/ExpectTerminated), removing the dependency on shutdown timing.

Evidence. Reproducer: 10 concurrent ConfigurationDatabase.Tests processes
against the one dev SQL Server — before, 10/10 processes failed (17-77 failures
each, 100% the ctor->MigrateAsync timeout); after, 3 consecutive runs 10/10
green, 395/395 each. Realistic shape (3x AuditLog.Tests + 1x
ConfigurationDatabase.Tests concurrent): first post-fix run still showed the
NotifyDispatcher leak, then 3 consecutive clean runs. Standalone: AuditLog.Tests
383/383, ConfigurationDatabase.Tests 395/395 (12s, unchanged from baseline),
SiteCallAudit.Tests 66/66. No orphan databases or stuck app locks left behind.

docs/test_infra/test_infra_db.md documents the ephemeral per-test-class
databases, the throttle and its env override, the timeouts, and the stray sweep.
 2026-08-17 04:08:45 -04:00
Joseph Doherty 71c7ed91c5 docs(register): close rows 81 (cap scoped to site_events via sampled page-layout model — dbstat proven unavailable and pinned) and 84 (backlog note once per tick); add rows 92-94 (mid-history width blindness, schema-calibrated constants, dbstat tripwire semantics) 2026-08-17 04:00:02 -04:00
Joseph Doherty 5162408a9f merge(agent-b): rows 81+84 — storage cap now measures site_events' own footprint (sampled page-layout model; dbstat probed unavailable and pinned); backlog note once per tick 2026-08-17 03:59:21 -04:00
Joseph Doherty ddb3c32e06 fix(siteeventlog): scope the storage cap to site_events' own pages (register rows 81, 84) 
Row 81 — the cap could not be enforced on a replicated node. MaxStorageMb was
compared against page_count-freelist_count over the WHOLE consolidated LocalDb
file, which since Phase 1/2 holds ten tables plus __localdb_oplog /
__localdb_row_version / __localdb_snapshot_state. Every 1000-row trim batch freed
~390 KB of site_events pages and added ~1000 oplog rows + ~1000 tombstones to the
same file, so the measured size did not fall, the "size did not decrease" guard
fired, and the trim stopped 2.5x over a 64 MB cap with 226 k rows still present.
A node whose peer was down could not enforce its cap at all, and the event log's
budget was in practice shared with bookkeeping it does not own.

The cap now measures site_events AND ITS FOUR INDEXES only, so replication
bookkeeping can neither consume nor block the budget — and the cap stops being a
function of the peer's health, which was the wrong direction of coupling: an
inflated oplog is a replication fault, and letting it drive event deletion would
destroy operational history in response to one.

dbstat — the exact per-table measurement the register proposed — is NOT usable
here, verified before implementing. It is a compile-time option and the bundled
SQLitePCLRaw.lib.e_sqlite3 (SQLite 3.53.3) does not enable it: PRAGMA
compile_options lists no ENABLE_DBSTAT_VTAB, pragma_module_list returns ten
modules with neither dbstat nor sqlite_dbpage, and selecting from dbstat fails
with "no such table". Switching SQLite builds was rejected — the provider is
shared with ZB.MOM.WW.LocalDb and every other consumer, so it trades cap accuracy
for fleet-wide version skew. The footprint is therefore modelled from sampled row
width (500 rows each end, octet_length so multi-byte UTF-8 is not under-counted)
through a page-layout model; whole pages rather than a flat rate, because page
fill collapses once rows approach half a page and a linear model underestimates
that by ~40%. Clamped by whole-file logical size, a true upper bound that can only
correct an over-estimate and never binds on a replicated node.

Index pages count against the cap deliberately: measured at ~34% of the table's
footprint, and the 1 GB default was sized against a 393 B/row figure that already
included them.

Guard semantics restated — two backstops replace the single on-disk check:
(1) row width unsampleable => refuse to trim (checked before the cap comparison,
since a zero width makes "over cap" unanswerable rather than false; the old loop's
answer to an unanswerable cap was to keep deleting); (2) estimated size not
falling across a batch => stop. (2) is the old guard re-based onto the
table-scoped metric, so it can no longer be tripped by replication bookkeeping and
now means exactly one thing: events are arriving as fast as the trim removes them.
For that to be observable the loop re-reads the row count each batch, while
bytes-per-row is sampled once per tick and held fixed — re-sampling per batch
would jitter the estimate as the trim eats the head of the table and could stall
the loop through guard (2), the same self-defeating measurement as row 81 itself.

Row 84 — LogReplicationBacklogNote moved from per-phase to once per tick, driven
by the shared PurgeBudget. A tick where retention AND the cap trim both deleted
used to emit two identical Information notes; the tick total is also the more
useful number beside an oplog spike.

Measured model error: -0.8% / +8.4% / +12.8% across representative workloads,
biased high on purpose (an early cap forfeits budget; a low one fails at its only
job). Independently cross-checks row 34's harness at 397.2 vs 393 B/row.

Fail-before: reverting the estimator to whole-file semantics fails 5 of the new
tests and reproduces the original signature exactly (remaining=2000
estimate=21725184 cap=11534336 — rows destroyed, cap still unmet); restoring
per-phase notes fails the row-84 test with "collection contained 2 matching items".

No config keys added or renamed, so no rig/deploy changes. Decision record:
docs/plans/2026-08-17-site-event-log-cap-semantics-decision.md

Tests: SiteEventLogging 104/104 (was 93), Host 520/520, solution build clean.
 2026-08-17 03:58:41 -04:00
Joseph Doherty 817a72108e docs(register): close rows 85 (premise corrected — ensure moved into the CreateInstanceActor funnel; not replication-specific, enable path was a second live instance) and 83 (shared DeploymentManagerStartupProbe gate); add rows 89-91 (connection-only deploy not applied on peer, no peer-deletion removal path, SiteScriptCompileCacheTests absolute-count fragility) 2026-08-17 03:53:36 -04:00
Joseph Doherty 2cae8773bf merge(agent-a): rows 85+83 — EnsureDclConnections moved into the CreateInstanceActor funnel (self-healed/enabled instances get their DCL connections); ReconcileStartsInstanceTests gated on first instance-count report 2026-08-17 03:52:35 -04:00
Joseph Doherty 524bdbbe90 test(siteruntime): gate ReconcileStartsInstanceTests on the DM's first instance-count report (register row 83) 
Row 67's suite still opened every test with a bare Task.Delay(500) after
constructing the Deployment Manager. Each test then writes its
deployed_configurations row -- and a row that lands before the boot-time load
reads storage is picked up by that load, which means the test passes while
proving the BOOT LOAD works rather than the startup-reconcile path it names. A
sleep only makes that race unlikely; on a loaded machine the load can still be
in flight when it expires and the test silently changes what it proves instead
of failing.

The row-78 fix session hit exactly this trap and closed it in
InstanceSweepSelfHealTests by gating on the first
ISiteHealthCollector.SetInstanceCounts report -- the edge the manager raises
unconditionally the moment its load completes, including for an empty site.
This applies the same gate here.

The probe is now a shared TestSupport helper (DeploymentManagerStartupProbe,
added with row 85) rather than a private nested class, so both suites use one
implementation.

Also pins InstanceSweepIntervalSeconds = 0 for this suite: with the sweep armed
its periodic self-heal would independently start the same instances and the
suite would stop distinguishing the row-67 mechanism from the row-78 one.

The remaining Task.Delay calls in the file are deliberate and left alone -- they
precede assertions that NOTHING happened (a disabled instance stayed stopped, a
redundant ensure-started changed nothing), which cannot be awaited on. The one
delay that was a readiness wait rather than an absence wait is now an
AwaitAssertAsync on the instance actually being routable.
 2026-08-17 03:48:12 -04:00
Joseph Doherty 8f2c2813ea fix(siteruntime): self-healed and newly-enabled instances now get their DCL connections (register row 85) 
An Instance Actor started by a self-heal came up with NO DCL connection actors,
so every tag it subscribed was answered "Unknown connection: <name>" by the DCL
manager indefinitely. Proven live 2026-08-16 by the row-78 close-out probe: the
sweep correctly started the replication-seeded instances, then every subscribe
failed for 6+ min until a second pair restart.

Root cause is narrower than the register assumed. A DCL connection actor is
never created from data_connection_definitions -- that table has no production
reader at all (GetAllDataConnectionDefinitionsAsync is called only by a test).
It exists only as a side effect of starting an instance, from the Connections
map inside that instance's own flattened config, via EnsureDclConnections.

That ensure was the CALLER's job and only two of the four callers made it:

  CreateInstanceActorBatch (boot)   ensured  OK
  ApplyDeployment (deploy)          ensured  OK
  StartStoredInstance               MISSING  <- row-67 ensure-started + row-78 sweep
  HandleEnableResult                MISSING  <- enabling a disabled instance

The two misses are the two paths added after the original pair -- the signature
of a convention living at call sites instead of in the funnel.

It was never replication-specific: the boot load reads only ENABLED configs, so
a node booting with an instance disabled never creates its connections either,
and enabling it afterwards hit the identical dead end on an ordinary
unreplicated site.

Fix: EnsureDclConnections moves into CreateInstanceActor, already documented as
the single Instance Actor creation funnel (it is where the running revision is
stamped for the same reason), and the two caller-side calls are removed. Placed
before the already-exists guard so the two paths that already called it keep
their exact previous semantics. No new message, option or DCL change -- it
reuses EnsureDclConnection's existing per-name hash guard, so steady state is
silent. Structural rather than another guard: a fifth start path cannot forget.

Deliberately NOT done, with rationale in the memo: a sweep pass over
data_connection_definitions. It would not have fixed row 85 (that table is empty
on a site that has only ever received instance deploys), and re-hashing from it
would fight the deploy path over the same per-name hash slot, recreating
connection actors and dropping live subscriptions on a loop. Two residual
sibling gaps (connection-only artifact deploy replicating to a peer that never
adopts it, and the mirror-image removal case) are recorded in the memo and the
component docs as open, not silently skipped.

Tests: SelfHealCreatesDclConnectionTests, 7 tests asserting on the
CreateConnectionCommand messages sent to a recording DCL stand-in. Verified
fail-before against a behavioral revert (call removed from the funnel, restored
at the two original sites): 4 fail / 3 pass, the 3 being steady-state silence
plus two refactor guards that must pass on both sides.

Docs: sweep design memo gains section 7; Component-SiteRuntime,
Component-DataConnectionLayer, docs/components/SiteRuntime and
docs/components/DataConnectionLayer updated (the deploy-flow step ordering in
docs/components/SiteRuntime was stale as written).
 2026-08-17 03:48:01 -04:00
Joseph Doherty 92b587d2a1 docs(register): record Low-tail sweep dispositions — close rows 58 (mutation-verified retention pins), 60 (premise corrected: DTOs not EF entities; lazy-loading pin), 65 (trigger fired — prune rides BundleSessionEvictionService); refresh rows 36/37/59/64 with re-verification evidence; add rows 86-88 (CLI/UI deployment scope divergence, CLI unbounded non-transactional scoping, silent --status parse fallback) 2026-08-17 03:44:27 -04:00
Joseph Doherty 3d94bec308 merge(agent-d): low-tail sweep rows 36/37/58/59/60/64/65 — retention pin + lazy-loading pin + serial-loop remark + row-65 prune tick on BundleSessionEvictionService 2026-08-17 03:43:08 -04:00
Joseph Doherty 2807b5a3ea fix(transport)+test: close deferred-work register rows 58, 60, 65 (Low-tail sweep) 
Row 65 — trigger FIRED, fixed. BundleUnlockAttempt rows were pruned only by
nodes actively serving unlocks, so a cluster that stopped importing entirely
left its last rows in place forever. The row's own trigger was "Transport gains
a central scheduler a prune tick could ride" — it has one now:
BundleSessionEvictionService, the central-only 1-minute session sweep added by
the transport-session fixes. It now also ticks the new
IBundleUnlockAttemptWindow.PruneAsync. Both drivers share the ONE 15-minute CAS
cadence claim, so ticking on a 1-minute sweep costs at most one bounded DELETE
per quarter hour and the two can never prune concurrently. PruneAsync is
contractually non-throwing (unlike TryRegisterAttemptAsync, which throws and
lets the limiter own the fail-open decision) — a background sweep has no caller
to report to and must survive a database blip. Both Transport docs updated;
three tests in FleetWideUnlockThrottleTests, incl. the retention sweep running
with no unlock traffic anywhere in the test, which is the whole point.

Row 58 — closed with a pin, and the OLD pin was found blind. The existing
Deployments_Dispose_UnsubscribesFromNotifier is behavioural: it is satisfied by
the _disposed guard alone, so deleting the `-=` from Dispose leaves it green
while the singleton notifier keeps a delegate pinning the circuit-scoped
component. Verified by mutation — with the `-=` removed the old test still
passed and only the two new pins failed. Added:
  * a structural retention pin asserting the notifier's invocation list actually
    shrinks to 0 on Dispose;
  * a source scan pinning the row's stated TRIGGER ("a second page subscribes,
    or any subscriber lands without a matching Dispose unsubscribe") — every
    CentralUI file that subscribes must also unsubscribe and be IDisposable,
    with a non-empty guard so the scan cannot pass vacuously.

Row 60 — closed with a pin, and the register's premise corrected.
GlobalArtifactSnapshot does NOT hold EF-materialised entities: it is projected
into pure Commons DTOs before the fan-out. The EF state that genuinely crosses
into the parallel send is the tracked Site entities the lambda closes over, and
that is safe only while lazy loading is off. Pinned in ConfigurationDatabase
.Tests two ways — no proxies extension on the built DbContextOptions, and no
EntityFrameworkCore.Proxies package reference anywhere in the repo (the
structurally stronger half: UseLazyLoadingProxies() is not a compilable call
without it). The real invariant is now spelled out at the fan-out itself.

Row 59 — no product change; added the missing tripwire remark stating the
constraint the safety argument rests on (serial by design, one scoped DbContext,
do not Task.WhenAll; if it gets slow the fix is a scope PER SITE, not a shared
scope under fan-out). Line numbers in the register have drifted to :528-545.

Rows 36, 37, 64: verification only, no change — see the sweep report.
 2026-08-17 03:42:31 -04:00
Joseph Doherty 07220bc655 docs(register): row-78 live-probe evidence (sweep self-heal fired in the original failure sequence); add row 85 — DCL connection actors not spawned for replication-seeded connection configs (row 78 one layer down, found by the probe) 2026-08-16 04:56:15 -04:00
Joseph Doherty 019edf1c46 test(playwright): row-28 follow-through — three dialog flows still expected the old 'Delete' label (SiteCalls Discard, Notification Discard, Instance Disable); labels went explicit in 0db253b5 but these text asserts/filters were missed because that wave's suite run predated the rig redeploy 2026-08-16 04:34:56 -04:00
Joseph Doherty b6676a89e6 docs(register): close rows 34, 62, 71-78, 80 with evidence; row 79 mitigation chosen (accept + runbook); add rows 81-84 (cap-vs-oplog self-fight, AuditLog MS SQL fixture parallel flake, boot-load test-gating trap, double backlog note) 2026-08-16 04:21:06 -04:00
Joseph Doherty f46cc19736 Merge rows 78+72: DeploymentManager singleton instance sweep — starts replication-seeded enabled instances and converges stale revisions through the real HandleDeploy path; one repair attempt per (instance, revision) 2026-08-16 04:17:45 -04:00
Joseph Doherty 4d39ad3cd7 Merge rows 80+73+71: nullable LocalDb health fields on unreplicated sites (Host supplies ReplicationIsConfigured), StartupReconcilerActive on the health report (proto field 29) with central No-self-heal badge, standby active-node ERR downgraded via narrowly-keyed Serilog filter 
# Conflicts:
#	CLAUDE.md
 2026-08-16 04:17:40 -04:00
Joseph Doherty 9f418a6aea Merge row 34: site_events purge burst measured (2.73M-row worst case = 2.7x MaxOplogRows) and paced — shared 250k/tick budget, 5-min catch-up, ceiling-hit counter 2026-08-16 04:17:16 -04:00
Joseph Doherty dd2391abe2 Merge small-fixes wave: rows 74-77, 62, 79-runbook (gauge delta asserts, Approve caution styling via DialogSeverity, CLI README sample, ActorCell sweep clean, ISiteEventLogger lifetime pin, OrbStack/virtiofs runbook) 2026-08-16 04:17:16 -04:00
Joseph Doherty e606b24a55 fix(siteruntime): instance reconcile sweep closes config/runtime divergence (register rows 78, 72) 
The site's deployed_configurations rows are the durable truth about what should be
running; the set of live Instance Actors is what IS running. Nothing kept the two in
agreement after boot, and nothing surfaced a disagreement.

Row 78 (availability, proven live 2026-08-15) -- deployed_configurations is one of
LocalDb's ten replicated tables, so on a replicated pair a config can appear in a
node's file with no local actor involved at all: the CDC sync applier writes the row
directly. Live sequence: site-a-b's reconcile fetched 4 configs at its solo boot; the
pair was then restarted with a fresh site-a-a file and snapshot replication seeded
site-a-a's table within ~5s -- AFTER its boot-time load had read an empty table, and
invisibly to its own reconcile (central rightly reported "0 fetched", because central's
view of the site was correct). Enabled instances sat dead with their configs present
until yet another restart. Row 67's EnsureInstanceStartedCommand only covers configs
the reconcile itself FETCHES; this is the same gap through a side door.

Row 72 (silent divergence) -- for a STALE ReconcileGapItem the pass writes the newer
config via StoreDeployedConfigIfNewerAsync, a pure file write, while the running
Instance Actor keeps executing the previously compiled config until the next restart
or redeploy. Nothing said "stored revision != running revision".

Both are closed by one periodic sweep owned by the Deployment Manager singleton
(ScadaBridge:SiteRuntime:InstanceSweepIntervalSeconds, default 60, validated >= 0;
0 disables). Each tick re-reads the durable rows off-thread and, on the actor thread,
for every ENABLED stored config either starts a missing Instance Actor (row 78) or --
when the running one's revision no longer matches the stored one -- re-applies the
stored config through the ordinary redeploy path (row 72).

A sweep was chosen over a LocalDb change hook deliberately: a hook covers only the
replication arrival path, and only on the half of the fleet running with replication
enabled, whereas the sweep is arrival-path-agnostic (replication seeding, post-boot
snapshot resync, a hand-restored file, any future path). It is also not available --
ILocalDb 0.3.0 exposes no change-notification or applied-rows callback, so a hook would
mean a new upstream capability in scadaproj plus an in-process bus into this singleton,
a cross-repo change to close a site-local gap.

Row 72's decision is to CONVERGE, not merely warn. The stored row is what central
successfully deployed; an actor serving the config central believes was replaced is the
actual fault, and a surface-only warning leaves the operator to issue by hand exactly
the redeploy the sweep would have performed. Reusing HandleDeploy means the compile
gate, warm-then-gate ordering, redeploy-supersede, the static-override / native-alarm
reset and all-or-nothing-per-instance are the production ones, not a second
implementation -- and a stop still does NOT cancel an in-flight script run, exactly as
for a central redeploy.

Guards, shared with row 67 via the extracted InstanceIsOwnedByADeployPipeline predicate
and the shared StartStoredInstance helper:
- a DISABLED instance is never started (config row, no actor is its correct state);
- the sweep defers to any deploy pipeline owning the instance (compile warm in flight,
  predecessor still terminating, deployment awaiting its persistence/init join), so it
  can never interleave with a real redeploy -- that window has the config stored and no
  actor, which an unguarded sweep would answer by creating a child whose actor name is
  still registered;
- inert until boot-time Instance Actor creation has finished, so a tick between
  staggered batches does not "heal" instances the boot load is about to create;
- ONE repair attempt per (instance, revision) pair -- this is what stops a periodic
  self-heal becoming a restart loop for an actor that dies during init, or for a stored
  config the compile gate rejects (a rejection leaves the running instance untouched:
  the gate returns before anything is stopped);
- reads never overlap, so a slow read cannot stack ticks into duplicate repairs.

A sweep that finds nothing is completely silent -- no log above Trace, no site event.
A repair emits the same Warning `deployment` site-event pattern as row 67 with the
trigger named ("deployment-manager sweep self-heal") so it is distinguishable from a
startup-reconcile heal; a stale-revision restart names BOTH revisions and a rejected
one adds an Error event. The synthetic redeploy replies to the singleton itself rather
than NoSender, because dead letters are a monitored site health metric.

CreateInstanceActor gains a revisionHash parameter and is the single creation funnel
that records it, so the stale check covers every path that can start an Instance Actor;
a missing entry reads as "unknown", never "stale", so a future path bypassing the funnel
degrades to no-op rather than to spurious restarts.

Fail-before by behavioral revert (sweep timer left unarmed): 5 of 10 new tests fail,
including both live symptoms -- the row-78 repro ("A replication-seeded config for an
enabled instance left no running Instance Actor") and the row-72 repro, whose snapshot
still shows the rev-1 value 111 while the store holds rev-2's 222. The 5 negative
guards (disabled / in-flight redeploy / matching revisions / silent-when-healthy /
interval-zero) are green in both states, as they must be.

SiteRuntime 626/626, Host 505/505, AuditLog 383/383.

Design note: docs/plans/2026-08-15-instance-sweep-self-heal-design.md
 2026-08-16 04:16:06 -04:00
Joseph Doherty d9a35846d9 perf(siteeventlog): pace the site_events purge burst — per-tick delete ceiling + accelerated catch-up (row 34) 
Measured the unbounded purge burst and found it can force a full ten-table
snapshot resync, then bounded it. No purge-exemption path was added — the
deletes are still CDC-captured like any other write, which stays ruled out
by design; what changed is how many a single tick performs.

MEASURED (harness: real EventLogPurgeService over a real LocalDb file with
site_events genuinely RegisterReplicated, plus a two-node loopback gRPC
replication pair; docker rig untouched):

  393 bytes/row incl. indexes => a 1 GB event log holds ~2,731,947 rows.
  Expiring all of them in ONE tick:  285 s, 2,731,947 oplog rows,
  284 MB oplog payload, 2.73 M row-version tombstones (7-day retention),
  i.e. 2.7x LocalDb's MaxOplogRows default of 1,000,000. Crossing that makes
  the 60 s maintenance tick prune the oplog and set needs_snapshot — a FULL
  snapshot of all ten replicated tables, bought by a routine retention purge.

  The resident backlog is produced-minus-drained, so a healthy peer blunts it:
  on loopback the same burst peaked at 391,424 and cleared in 6.0 min. That is
  the problem, not the reassurance — the old safety margin was a function of
  peer throughput and vanished with the peer down. A bounded tick is safe at
  any peer speed.

  Drain is not a bottleneck: 7.6k-19.8k tombstones/s (20k / 500k / 2.73M runs).

SHIPPED:
  - SiteEventLogOptions.MaxPurgedRowsPerTick (default 250,000; 0 = unlimited
    escape hatch, negatives rejected). One SHARED budget across the retention
    purge and the cap trim — two independently-capped phases would let a tick
    emit twice the intended burst. 4x margin under MaxOplogRows, which is also
    shared with every other replicated table's ordinary traffic.
  - SiteEventLogOptions.PurgeCatchUpInterval (default 5 min, validated > 0 and
    <= PurgeInterval). Without it pacing would make retention converge in 11
    DAYS at the daily cadence; with it the worst case converges in ~11 ticks
    (~55 min), every burst bounded. 5 min rather than 1 so ticks cannot outrun
    the peer's drain and re-accumulate the backlog.
  - A ceiling is claimed only when work genuinely remains (bounded EXISTS probe),
    so an expired set landing exactly on the budget does not arm a no-op retry.
  - Visibility: rate-limited (15 min) Warning naming the ceiling and the
    accelerated interval, plus an unthrottled counter
    scadabridge.site.event_log.purge_ceiling_hits — its SHAPE distinguishes a
    backlog converging from a site whose event rate outruns one paced tick.
  - The purge loop is a delay loop rather than a PeriodicTimer, so its cadence
    now anchors to the END of the previous purge. Documented.

ROW-33 INTERPLAY VERIFIED: pacing lowers each spike and lengthens total purge
duration, but a full 250k tick drains in ~23 s (~4 min even at 10x a real
link's penalty) against the 15-minute sustained dwell and a 5-minute gap to
the next tick. The UNPACED 2.73M burst had only 2.5x of margin (6.0 min), so
pacing makes the badge assumption robust rather than merely preserving it.

Docs travel together: Component-SiteEventLogging.md, topology-guide.md
("Reading the replication backlog" + the row-33 badge note), CLAUDE.md's
purge-burst sentence, and a dated measurement/decision note under docs/plans/.

Fail-before: the new behavioural tests do not compile against the pre-fix
service (RunPurge was void); with PurgeBudget neutralised to always-unlimited,
6 of the new tests fail and all 19 pre-existing ones pass.

Tests: SiteEventLogging 93/93, Commons 715/715, HealthMonitoring 115/115,
Host 505/505. Solution build clean.
 2026-08-16 04:05:21 -04:00
Joseph Doherty 8b761841bc fix(logging): row 71 — stop logging the standby's by-design active-node verdict at ERROR 
A healthy standby logged "Health check active-node with status Unhealthy …
'not the site pair primary (standby)'" at Error on every probe — every 10s on a
site node, and (discovered while fixing this) every 5s on a central standby,
which Traefik health-probes at that cadence. Correct-by-design state at an
alarming severity: it floods the log and trains operators to ignore ERR.

Mechanism: DefaultHealthCheckService logs one line per check per probe and picks
the level from the verdict — Debug/Warning/Error for Healthy/Degraded/Unhealthy.
Both active-node checks answer Unhealthy on the standby because 503 IS the
contract /health/active publishes. There is no IHealthCheckPublisher in this
repo, so each line is one inbound HTTP probe.

Fix: one Serilog exclusion filter in LoggerConfigurationFactory. A level
override could not work — Serilog's MinimumLevel.Override only RAISES a
per-source floor, and a LogEvent's Level is immutable, so neither an override
nor an enricher can rewrite Error as Information. Changing the check to report
Degraded was rejected: it would alter the HTTP status that /health/active
publishes, which is load-bearing for Traefik and the family dashboard.

The constraint (genuine failures must still log at Error) is preserved by keying
the predicate on the check NAME rather than the logger category: localdb,
akka-cluster, database and required-singletons keep their Error lines, and the
active-node check has no error path of its own — the active node always answers
Healthy — so the only event this can ever suppress is a standby reporting that
it is a standby. The verdict stays fully observable via /health/active's own
HTTP status, the readiness probe's cluster view, and the health dashboard.

Also corrects a stale CLAUDE.md claim found while verifying what consumes this
check: "/health/active is central-only (site nodes map no /health/* at all)" is
wrong — SiteServiceRegistration registers akka-cluster + localdb + the
role-scoped SitePairActiveNodeHealthCheck, and Program.cs's Site branch calls
MapZbHealth() on the HTTP/1.1 listener.

Fail-before evidence: with the filter behaviourally disabled,
StandbyActiveNodeVerdict_IsNotLogged fails "Collection was not empty" while the
other 9 tests in the file — the must-still-log constraints — pass either way,
which is what proves the filter is narrow rather than a blanket suppression.

Tests: Host 518 green (10 new).
 2026-08-15 21:41:01 -04:00
Joseph Doherty a2f0086fa1 fix(health): rows 80+73 — unconfigured replication reports null, reconciler presence surfaced 
ROW 80 (observability correctness). An unreplicated site published
LocalDbReplicationConnected=false / LocalDbOplogBacklog=0, contradicting the
nullable semantics every doc and the proto already state ("null means no
reading, NOT disconnected with an empty backlog").

Root cause: the replication engine is registered unconditionally and idles
without a peer, so ISyncStatus answers false/0 on a node that was never meant to
replicate, and LocalDbReplicationStatusReporter passed that straight through.
Two consequences: an operator read connected:false on a healthy site as a fault,
and central's null branches — row 33's conditional KPI emission and
sustained-badge anchor exclusion — never received the null they exist for, so
rig site-b/site-c emitted a permanently misleading
localDbReplicationConnected=0 series.

Fix mirrors the SiteEventLogReplicationCheck precedent: the Host evaluates
SiteLocalDbSetup.ReplicationIsConfigured ONCE and passes it to
AddLocalDbReplicationHealthBridge as a third provider; unconfigured
short-circuits the probe to (null, null). HealthMonitoring never reads LocalDb
config itself, and the same captured value now also feeds
SiteEventLogReplicationCheck so the two surfaces cannot disagree.
ISiteHealthCollector.SetLocalDbReplicationStatus widens to bool? — the
non-nullable bool was the type-level root cause (null was inexpressible).
false now means exactly one thing: configured, peer session down = a real fault.

Wire/central/UI needed no change: the proto already carries BoolValue/Int64Value
wrappers and the aggregator + KPI source already branch on null. This makes code
match doc, so no doc claim was weakened.

Watch the trap that caused this: configured-ness is PeerAddress OR ApiKey (only
the dialling half sets a peer address). The pre-existing SiteLocalDbWiringTests
fixture sets ApiKey and IS a replicating node, so its false/0 assertion is
correct and is kept, with a comment saying so; the new
UnreplicatedSiteLocalDbHealthTests fixture is the genuinely-unreplicated one.

ROW 73 (silent loss of self-heal). AkkaHostedService skips
SiteReconciliationActor when no IDeploymentConfigFetcher is resolvable, and that
skip was a lone boot Warning — a node with no self-heal path at all went on
reporting itself perfectly healthy forever.

Adds tri-state SiteHealthReport.StartupReconcilerActive (null = not reported,
i.e. the central self-report or a pre-row-73 site build; true = running;
false = booted without it), latched once at boot by the Host in both branches.
Additive proto field 29 (BoolValue, vendored C# regenerated via
docker/regen-proto.sh centralcontrol; csproj restored, verified empty diff).
The skip branch also raises its log to Error — no configuration makes the
fetcher legitimately absent on a site node — and writes a best-effort
deployment/Error site event so the evidence outlives the boot log.

Central treatment: a "No self-heal" WARNING badge on the site tile, not a
failure. Config still reaches the node via LocalDb replication while it is up;
what it lost is recovery from a deploy that landed while it was down — degraded,
not offline. Only the explicit false is badged (badging null would light up the
whole fleet the moment central is upgraded ahead of the sites), and offline
sites are skipped, matching the replication-backlog badge.

Fail-before evidence: with the row-80 gate behaviourally reverted,
UnreplicatedSite_HealthReport_ReportsNullForBothLocalDbFields fails
"Expected: null / Actual: False" and 3 of the 8 new reporter unit tests fail.

Tests: HealthMonitoring 128, Host 518, CentralUI 1051, Communication 717,
Commons 715, SiteEventLogging 81 — all green.
 2026-08-15 21:40:48 -04:00
Joseph Doherty 851b528ca3 docs(deployment): row 79 — OrbStack/virtiofs rig hazard + proven recovery runbook 
Documents the host-level hazard behind row 69's LocalDb corruption, all
live-verified 2026-08-15. Scope is the macOS docker rigs only — this is a
property of the filesystem the rig's data files sit on, not a ScadaBridge defect
and not something a production Linux deployment is exposed to.

Every rig data file (site-localdb.db, auditlog.db, secrets) rides a relative
bind mount out of the repo tree, which on macOS crosses virtiofs — and virtiofs
does not give SQLite true fsync durability. `--force-recreate`, which deploy.sh
uses, is exactly the abrupt mid-write kill that turns that into a torn file.

Records both observed failure modes, and specifically how differently they
present, which is the part that makes this hard to spot: site-a-b failed loudly
while running (every sync call throwing, connected:false, backlog 892), while
site-a-a was corrupt LATENTLY and only became fatal on the next upgrade, when a
new build's boot-time re-registration read pages the old build never touched
(SIGABRT crash-loop x12). A clean-looking node proves nothing about its file.
Second mode is the VM idle-sleeping mid-operation — `vmgr ... msg=sleep` in
~/.orbstack/log/vmgr.log, then virtiofs `refresh_nodeid: race with new lookup`
errors and a hung docker until `orb restart docker` (data intact, containers
auto-restart). Written so a future session checks vmgr.log BEFORE investigating
ScadaBridge, which is where ~20 minutes went last time.

Recovery runbook is the one that actually worked: stop the pair together,
quarantine site-localdb.db AND its -wal/-shm sidecars with a .corrupt-<date>
suffix (never delete — forensics; and a stale WAL beside a fresh database is its
own corruption), restart together, then VERIFY convergence rather than assuming
it. Notes the precondition that this is a replicated pair's recovery — it leans
on the startup reconcile plus CDC snapshot resync, so an unreplicated node has
no peer to reseed from and needs a backup instead.

Mitigation options tabulated with their real costs (named volumes = native ext4
but loses the host-side inspectability the debugging sessions lean on;
disabling OrbStack idle-pause = a host setting that cannot be version-controlled
and silently does not exist on a new machine; accept + runbook = current), so
the status quo reads as a choice rather than an oversight.

Docs only — no compose change, no host config change, rig untouched.
 2026-08-15 21:34:17 -04:00
Joseph Doherty 40944a0d04 feat(centralui): row 75 — Approve is caution amber, not destructive red 
The Secured Writes page rendered Approve and Reject as IDENTICAL red buttons.
Row 28 fixed the labels but left the styling, and the styling was the worse
half: at the moment of a two-person-control decision the colour carried no
discriminating information at all, and the affirmative half of the control wore
the delete affordance.

Root cause is that `danger` was a two-state flag for a three-state concept, so a
consequential-but-not-destructive action had to choose between looking like a
delete and looking like the safe default. Adds the missing middle rung:

    DialogSeverity { Normal -> btn-primary, Caution -> btn-warning,
                     Destructive -> btn-danger }

DECISION. Approve -> Caution (amber): approving executes a real write to live
plant equipment, so it is not a benign primary action, but it is not destroying
anything either. Reject KEEPS Destructive (red): it is terminal and irreversible
for the request — the Operator must resubmit — and holding it red is what keeps
the two opposite decisions visually distinct now that Approve is amber. Dropping
Reject to neutral was considered and rejected: it would invert the bias, making
declining look like the safe default. Colour is never the only signal — both
buttons carry their verb.

Only Destructive inherits the row-28 "Delete" label fallback; an unlabelled
Caution reads "Confirm", since a caution action is by definition not a delete —
otherwise row 28 would come back through the new door.

ADDITIVE OVERLOAD, deliberately. The boolean form stays the signature for all
~34 existing call sites, which are genuinely two-state, so nothing in src/
outside SecuredWrites.razor changed. Adding a fifth optional parameter to the
boolean form instead would have silently un-matched every existing four-argument
NSubstitute stub — the exact row-28 gotcha (null Task, caller NREs, no compile
error). The overload cannot: the only breakage was six hand-written IDialogService
fakes failing to compile, which is the loud failure mode. That gotcha did land
once, in SecuredWritesTests' own stub, and is now stubbed on both overloads with
a comment saying why.

DialogState's `bool Danger` became `DialogSeverity Severity` rather than gaining
a second field — it is read only by DialogHost, and one field cannot disagree
with itself.

Fail-before, both halves:
  - Approve call site reverted to `danger: true` -> both new Approve pins fail.
  - Caution remapped to btn-danger in DialogHost -> the amber styling pin fails.

Tests: DialogSeverityStylingTests renders the REAL DialogService behind a real
DialogHost (FailoverConfirmLabelTests' harness), so it pins what the user sees,
not what an argument was — all three severities, both label fallbacks, and the
boolean adapter. SecuredWritesTests pins that the page asks for Caution, with a
negative control that it did NOT also go through the boolean form (without which
the assertion would pass on a call site left at danger:true). Reject's red is
pinned too, so a later "make this page less alarming" pass cannot flatten both
to one colour.

No Playwright selectors needed updating — the suite has no Secured Writes
coverage, and no `.btn-danger` selector elsewhere targets this page.

Docs: Component-CentralUI.md gains the severity contract under Technology (next
to the row-28 label contract) plus the decision on the Secured Writes section.

CentralUI.Tests: 1055/1055 green.
 2026-08-15 21:33:08 -04:00
Joseph Doherty 9d668861ad test(host): row 62 — pin ISiteEventLogger's Singleton lifetime in the site graph 
Six SiteRuntime call sites resolve ISiteEventLogger straight off the root
IServiceProvider with no scope: AlarmActor (cached in a field for the actor's
lifetime), ScriptActor (x2), InstanceActor (x2), NativeAlarmActor,
DeploymentManagerActor, and Scripts/ScriptRunLauncher. All correct today, and
correct for exactly one reason — the registration is AddSingleton, so a root
resolve yields what a scope would.

That makes the lifetime a load-bearing invariant with no enforcement anywhere.
If it ever changes, all six silently become register row 45, and nothing in the
tree notices: ValidateOnBuild walks CONSTRUCTOR graphs and these are imperative
GetService calls in method bodies; ValidateScopes would fire, but at runtime on
a site node on whichever path logs an event first, not at boot and not in CI;
CaptiveDependencyGuard catches the AlarmActor field case by design but cannot
see the five per-call resolves at all. This pin is the tripwire, and its failure
message names the six call sites that must be fixed alongside any such change.

Asserts the REGISTRATION, not the call sites — the SiteRuntime actor files are
correct as written and are untouched here.

Two facts, not one, because the interface registration is a FACTORY
(AddSingleton<ISiteEventLogger>(sp => sp.GetRequiredService<SiteEventLogger>())):
a Singleton lifetime on the interface alone does not prove callers share an
instance, since downgrading the concrete SiteEventLogger would make that
singleton factory resolve a scoped service from the root.

Fail-before, both halves, by temporarily flipping the real registration:
  - AddScoped<ISiteEventLogger> -> first pin FAILS, "Actual lifetime: Scoped",
    with the full row-62 pointer in the message.
  - AddScoped<SiteEventLogger> (interface left Singleton) -> the first pin stays
    GREEN and only the second catches it. That is the evidence the second fact
    is load-bearing rather than redundant.
Registration restored; both green.

Registration-only inspection — the provider is never built, so the pin costs
~30 ms and opens no SQLite file.

Host.Tests: 507/507 green.
 2026-08-15 21:24:35 -04:00
Joseph Doherty 4ae57bdb6b docs(cli): row 76 — audit-config query JSON sample shows afterStateJson + bundleImportId 
The `entries` array is a serialized `AuditLogEntry`, so both fields are always
on the wire; the README sample showed neither, which read as if the payload
were the seven columns the table projection happens to keep. Row 57 added that
projection and the table section already names both omitted fields — the JSON
sample above it was the stale half.

Added both to the sample and documented their nullability, which is the part a
script author actually needs: `afterStateJson` is the post-change snapshot
embedded as a string and is null for deletes (no after-state), `bundleImportId`
is the import session id when the change came from `transport import` and null
for a direct change. Also grew the sample from one entry to two so it now
mirrors the table-output sample directly below it row for row — the same 412
Template/Created and 413 ExternalSystem/Deleted pair — which is what makes the
"the table drops these two fields" sentence checkable against the JSON above it.

Docs only.
 2026-08-15 21:17:34 -04:00
Joseph Doherty 98c9458830 test(transport): row 74 — gauge asserts become delta-over-scoped-action, not absolute 
BundleSessionObservabilityTests' two gauge facts asserted absolute values
(Assert.Equal(0, baseline), Assert.Equal(2, ReadGauge())) against what is a
single process-global provider slot — the same latent shape as the row-32
QueueDepthGaugeTests flake, correct only while this file is the slot's sole
registrant. Converted to the delta-over-scoped-action pattern the row-66
rejection-counter facts in the same file already use: a new ReadGaugeDelta
helper measures how far the instrument MOVED across an action driven by this
test's own store, so an inherited non-zero baseline cancels out.

Two assertions got stronger in the process. The post-clear check was
Assert.Equal(0, ReadGauge()) — "the instrument reports zero", which any
unregistered slot satisfies; it is now "opening more sessions on the detached
store moves the gauge by nothing", which is the actual claim. And the
compare-and-clear fact gained a negative control: the superseded store's opens
must NOT reach the instrument, so a slot still pointing at oldStore can no
longer satisfy the assertion by coincidence when both stores hold equal counts.

Fail-before (temporary probe, not committed): with a foreign provider returning
7 registered before the service starts, the OLD body fails
`Assert.Equal(0, baseline)` with `Expected: 0 / Actual: 7`; the NEW delta body
passes the identical scenario.

Honest residual documented on the helper: a gauge has one slot and
last-writer-wins, so a registrant that clobbers it BETWEEN the two reads is
unrecoverable read-side — that needs row-32's mutual exclusion. It cannot arise
today (this is the only file in the assembly constructing a
BundleSessionEvictionService, and test assemblies are separate processes); a
second such file should bring the collection treatment with it.

Transport.Tests: 180/180 green.
 2026-08-15 21:16:55 -04:00
Joseph Doherty 3dc38fb674 docs(loadtest+register): close row 29 — 1-hour sustain run settles F1 as not-a-leak (working-set slope -28.89 MB/min, one gen-2 collect, 135M updates at 100.0% nominal); raw metrics JSON archived 2026-08-15 20:27:09 -04:00
Joseph Doherty e3ca77a234 docs(register+decision): rows 69/70/78/79/80 — site-a LocalDb corruption recovered, auto-down deprecation drill-verified benign (explicit provider-class wins; re-drill on every Akka bump), replication-seeded configs start no actors (row-67 side door), OrbStack virtiofs hazard, non-null localDb health fields on unreplicated sites 2026-08-15 19:26:37 -04:00
Joseph Doherty caf9f45165 docs(register): close rows 27, 28, 32, 33, 35, 40, 54-57, 61, 63, 66, 67 with evidence; add rows 71-77; record the row-42 seven-script audit result 2026-08-15 18:03:27 -04:00
Joseph Doherty 744ff36a7b merge: startup reconcile starts the healed InstanceActor (register row 67) — EnsureInstanceStartedCommand to the DM singleton, idempotent, disable-wins, Warning site event on self-heal 2026-08-15 17:56:21 -04:00
Joseph Doherty e8a90f18dc merge: external-system retry policy end-to-end (register row 27) — MaxRetries/RetryDelay now travel artifact→gRPC→site schema→S&F enqueue, CLI --max-retries/--retry-delay-seconds, ArtifactDiff compares them 2026-08-15 17:56:21 -04:00
Joseph Doherty 230c841d35 merge: tag-resolution counts fixed on first subscribe (register row 55) — HandleSubscribeCompleted was the missing fourth UpdateTagResolution push site, so a perfectly healthy connection reported 0/0 forever 2026-08-15 17:56:14 -04:00
Joseph Doherty c26b52ad68 merge: observability (register rows 33, 54, 56, 63, 66) — LocalDb backlog KPI history + sustained-threshold badge, honest stream-subscriber gauge rename + real data-connection gauges, central metrics port on the rig, ASPNETCORE_ENVIRONMENT on site containers, import-session rejection counter 2026-08-15 17:56:14 -04:00
Joseph Doherty d607011e5e merge: small fixes (register rows 28, 35, 57) — confirm dialogs name their own action (9 mislabeled 'Delete' buttons fixed incl. secured-write Approve/Reject), DeploymentRecords composite (DeployedAt DESC, Id DESC) index, audit-config query CLI table projection 2026-08-15 17:56:09 -04:00
Joseph Doherty acc6209665 merge: test-infra hardening (register rows 32, 40, 61) — queue-depth gauge collection isolation, wall-clock sleeps replaced with observable edges + negative controls, _injectedRepository test ctors deleted with empty guard allowlist 2026-08-15 17:56:09 -04:00
Joseph Doherty 92fe5acca7 refactor(auditlog+sitecallaudit): delete the _injectedRepository test ctors — register row 61 
AuditLogIngestActor and SiteCallAuditActor each exposed constructors taking a scoped
repository and caching it on a field for the actor's lifetime. Safe only because no
production call site picked those overloads — i.e. one call site away from re-introducing
the row-45 captive-dependency bug — which is why CaptiveDependencyGuard had to allowlist
both fields.

Preferred fix applied: the test fixtures fold onto the IServiceProvider overload with a
stub provider, and the constructors are gone.

Product deletions:
  - AuditLogIngestActor: the (IAuditLogRepository, ILogger) ctor, the _injectedRepository
    field and its branch in OnIngestAsync, and OnCachedTelemetryWithoutDualWriteAsync (the
    "this ctor cannot dual-write" fallback handler that only that ctor installed).
    _serviceProvider is now non-nullable and the null-forgiving operators are gone.
  - SiteCallAuditActor: both test ctors (repo-only, and repo + reconciliation
    collaborators), the _injectedRepository field and its four branches, and the private
    SerializedRepository class. _backgroundTimersEnabled went with them: only the deleted
    repo-only ctor ever set it false, so it was a constant and its two guards are removed.
    The reconciliation tick is still gated — on the collaborators being resolvable, which
    is how production reaches the same state.

Test fixtures (12 call sites): two small TestSupport helpers build the stub providers.
SiteCallAuditActor's production wiring resolves a FRESH repository per scope, so the
serializing decorator existed only to protect fixtures that share one DbContext-backed
instance across every scope — it moves into the test project as
SerializingSiteCallAuditRepository, which is where the hazard is actually created.
Fixtures that relied on it keep it. The relay test's ICentralAuditWriter now arrives via
the provider, exactly as production resolves it.

Behaviour is preserved on the paths under test: the deleted audit ctor passed
redactor:null / failureCounter:null, and a provider registering neither resolves both to
null, so the SafeDefaultAuditRedactor fallback and counter-free logging are unchanged. No
fixture on the folded sites sends IngestCachedTelemetryCommand, so swapping to the real
dual-write handler is inert. The SiteCallAudit purge timer now arms in fixtures that
previously had timers off, but every one of them uses default purge settings whose first
tick is 5 minutes out — nothing fires inside a test.

One reconciliation test was renamed and rewritten rather than deleted:
TestCtor_RepositoryOnly_DoesNotStartReconciliationTick became
NoReconciliationCollaboratorsRegistered_DoesNotStartReconciliationTick — the same
guarantee, expressed the way production actually reaches it.

Guard: the allowlist is now EMPTY. The staleness self-test (every entry must still match a
flagged field) is replaced by two stronger pins — Allowlist_IsEmpty asserts nothing is
exempt at all, and FormerlyAllowlistedActors_NowScanClean scans both actors with the
allowlist DISABLED, so the empty allowlist is evidence the fields are gone rather than
evidence the scan stopped looking.

Negative control: re-adding an ISiteCallAuditRepository field to SiteCallAuditActor fails
both FormerlyAllowlistedActors_NowScanClean and the real
Central_NoActorHoldsCaptiveScopedState scan.

Verification: solution builds clean. AuditLog.Tests 3/3 (383), SiteCallAudit.Tests 3/3
(66), Host.Tests 2/2 (505), IntegrationTests 2/2 (115), Communication.Tests 2/2 (717).
 2026-08-15 17:49:39 -04:00
Joseph Doherty 7a7d345b89 test: replace wall-clock sleeps used as synchronization — register row 40 
Six sleeps across four test files stood in for an observable edge. Each preceded an
ABSENCE assertion (or "settled" before one), so they failed toward false-pass rather than
flake: the assertion could not distinguish "the mechanism ran and correctly stayed quiet"
from "the mechanism had not run yet".

ScriptDeadlineAtEnqueueTests (two sleeps, the worst offender)
  - :105 Thread.Sleep(2s) — now two explicit edges: the run must be observably QUEUED
    (scheduler.QueueDepth >= 1, never checked before), then the wait polls until the 1 s
    budget has elapsed, bailing early if the run leaves the queue, and asserts it is STILL
    queued afterwards. The test now proves the design claim it was named for — queue wait
    burns the run's budget — instead of assuming it.
  - :128 Thread.Sleep(2s) + Assert.Equal(0, DetachedThreadCount) — replaced by a positive
    control: a second script that genuinely wedges a worker, on the same scheduler, same
    1 s timeout + 1 s grace, armed strictly later. Its stuck report is the edge; once it
    lands, every watchdog armed at or before it has provably finished its grace delay. The
    absence is then asserted per script name, and the detach counter is pinned at 1 (the
    wedged run) rather than 0 — which a counter that never moves also satisfied.

DeploymentWarmThenGateTests (:117 plus two sibling Task.Delay(500) startup waits)
  - The manager calls SetInstanceCounts unconditionally when its startup config load
    completes, so a report counter on the test collector is the startup edge. The
    empty-site zero is now the manager's own observation instead of an untouched field.

SiteAlarmLiveCacheServiceTests (:195, :216)
  - Both sampled IsLive once after a fixed sleep. Now they poll continuously across three
    linger windows on a new teardown observable (the mock client counts subscription
    cancellations), so a stop at any instant inside the window is caught, and each closes
    with a positive control proving that counter moves.

DataConnectionActorTests (:435)
  - A Task.Delay(200) "settle" after an AwaitCondition on a connect counter. The counter
    only proves ConnectAsync was CALLED; the actor is still Connecting until it processes
    the piped result. Now waits for the Connected health publication that BecomeConnected
    emits, so the subscribe under test provably runs against Connected rather than being
    silently stashed. (The stash path keeps its own coverage in
    WP6_ConnectingState_StashesSubscribeRequests.)

Negative controls (product broken in the working tree, old vs new test compared):
  - Stuck-script watchdog disabled entirely: OLD test passed 2/2, NEW test failed. The
    whole watchdog could be deleted and the old assertions were satisfied.
  - Startup reporting a phantom instance count and taking longer than 500 ms: OLD passed
    2/2, NEW failed (expected 0, actual 1).
  - Linger cancellation disabled + the remaining-viewer guard dropped, so the stream is
    torn down under a live viewer while IsLive keeps reporting true:
    Resubscribe_Within_Linger OLD passed 2/2, NEW failed. (Last_Viewer_Leaving caught this
    particular break in the old form too, via a later assertion — its strengthening is the
    teardown observable and the positive control, not a false-pass fix.)
  - DataConnectionActorTests :435 has no false-pass demo: the sleep guarded no assertion
    there, and the change is determinism plus proving the claimed state.

Also fixes a latent race these timing changes exposed in
InstanceActorChildAttributeRaceTests: it resolved child actors single-shot immediately
after construction, but ResolveOne throws until the child's cell is published on the
parent AND the cell's Actor property stays null until the child's constructor has run on
its own mailbox thread. Neither is ordered against the test thread, so it failed both ways
under full-suite load (~1 run in 6 once the sleeps above stopped padding the schedule;
14/14 green on the unmodified tree, so it was latent, not new). Resolution now retries to
a bounded deadline; the assertion is unchanged.

Verification: SiteRuntime.Tests 8/8 green (605 tests), Communication.Tests 3/3 (717),
DataConnectionLayer.Tests 3/3 (319). Test-only change — no src/ modifications.
 2026-08-15 17:06:51 -04:00
Joseph Doherty b2cab59cb9 feat(cli): compact table projection for audit-config query (register row 57) 
The command passed no `tableProjector`, so `--format table` handed the renderer
an `AuditLogQueryResult` OBJECT. That takes the generic Property/Value branch,
which printed the entire `entries` array as raw JSON inside a single cell. JSON
output was and remains correct, and is the primary format for this command.

Adds `AuditConfigTableProjection`, mirroring `TemplateTableProjection`:
string->string over the raw body, applied only on the table path, returning the
input verbatim when it is not JSON or not a recognised shape so the renderer's
own fallbacks still apply. It unwraps the envelope to a row array — id,
timestamp, user, action, entityType, entityName, entityId — and drops
`afterStateJson`, the full post-change entity snapshot, which is the audit
analogue of template's fat child arrays: one per row fills a terminal.
`bundleImportId` is dropped as mostly-null noise.

`totalCount` would be lost by a plain unwrap, so a truncated page gains a single
trailing row with a `note` cell ("showing 50 of 137 — use --page / --page-size
for the rest"). When the page IS the whole result set the column is not emitted
at all, so a small query's table stays exactly as narrow as its data, and the
note row carries no `id` so it cannot be mistaken for an entry.

A `--detail` flag skips the projection and renders the raw payload, matching
`template list` / `template get` — that plus `--format json` is how to reach
`afterStateJson`. A bare array root is also accepted, so the projection does not
regress if the payload is ever unwrapped upstream or the deprecated `audit-log`
alias meets an older server.

Tests build their fixtures by serialising the REAL AuditLogQueryResult /
AuditLogEntry types, so renaming a server property fails the test instead of
silently blanking a column. CLI README + Component-CLI.md document the table
shape, the omitted fields, and --detail.
 2026-08-15 13:22:41 -04:00
Joseph Doherty be29bf471e perf(configdb): index DeploymentRecords on the deployment-list sort key (register row 35) 
The deployment list orders `DeployedAt DESC, Id DESC` and joins `Instance` to
make SiteId filterable, but the only DeployedAt index was single-column and
ascending, so the server sorted the filtered set on every page request.

Adds `IX_DeploymentRecords_DeployedAt_Id (DeployedAt DESC, Id DESC)` — the exact
sort key, both columns descending, so a forward scan feeds the paged Skip/Take
directly. Id is in the key as the deterministic tie-break: a bulk site deploy
stamps many rows in the same DeployedAt instant, and without it a row can repeat
or vanish across page boundaries. Three reads share the key
(QueryDeploymentListPageAsync, QueryDeploymentSummariesAsync, and the single-row
GetCurrentDeploymentStatusAsync).

DROPS the old `IX_DeploymentRecords_DeployedAt` rather than keeping both. It is
strictly redundant, not merely overlapping: DeployedAt leads both, so every
predicate it served the composite serves too (direction does not affect
seekability — SQL Server scans either way), and the leaf rows are the same width,
because Id is the clustered PK and was therefore already in the old nonclustered
index's leaf as the row locator. All that changes is Id becoming a declared,
ordered key column. Keeping it would maintain a second index on every deployment
insert and status transition for no read benefit.

Honest caveat: because the clustering key is implicitly ordered within a
nonclustered index, the optimizer may already have been able to serve this
ordering by scanning the old index backward. The explicit composite makes it
deterministic and self-documenting rather than dependent on that inference, and
costs nothing to declare.

Migration 20260815165452_AddDeploymentRecordListSortIndex (scaffolded with a real
build, non-empty, snapshot updated) plus the hand-written production script under
docs/plans/sql/ following the AddAuditLogAndNotificationCoveringIndexes
precedent. SystemArtifactDeploymentRecords has its own separate DeployedAt index
and is deliberately untouched.
 2026-08-15 13:22:29 -04:00
Joseph Doherty 0db253b55f fix(centralui): confirm dialogs name their own action instead of always saying "Delete" (register row 28) 
The Health dashboard's "Trigger failover" confirmation offered a red button
labelled *Delete*. The copy was right; the button was not. `DialogHost` derived
the confirm label solely from `danger`, so `danger: true` meant the verb was
literally "Delete" for every caller — and no caller could say otherwise, because
`IDialogService.ConfirmAsync` had no label parameter.

`danger` now selects styling only; a new optional `confirmLabel` names the action
and wins over the fallback. The kind/danger fallback is deliberately RETAINED
("Save" for a prompt, "Delete" for a bare danger confirm, "Confirm" otherwise):
around 20 call sites are genuine deletes where it is correct, and the Playwright
suites click the literal "Delete". Requiring the label everywhere would have
churned those 20 sites for no behaviour change, with the miss-detection resting
on Playwright runs that need a live rig.

Audited all 34 confirm call sites; 9 read "Delete" for something that is not a
delete and are relabelled:

  Fail over  CentralFailoverControl (central pair, site pair)
  Approve    SecuredWrites — approve (this one WRITES TO A DEVICE)
  Reject     SecuredWrites — reject
  Disable    Topology — disable instance
  Discard    SiteCallsReport, NotificationReport, ParkedMessages (bulk + single)

The remaining 25 are correct as they stand: 18 genuine deletes plus 7 non-danger
confirms ("Revert to base", the three Retry paths) whose button reads "Confirm".

Tests: FailoverConfirmLabelTests drives the real DialogService behind a real
DialogHost — a substitute would pin the argument but not what the operator reads
— covering both branches, that the button keeps btn-danger, and that an
unlabelled danger confirm still falls back to "Delete". Note its click is
deliberately not awaited: with a real dialog the handler parks on ConfirmAsync
until answered, so awaiting deadlocks. Existing IDialogService fakes and
NSubstitute matchers gain the 4th parameter (an optional param is baked into the
call site, so a 3-arg Arg.Any stub silently stops matching a labelled call).
 2026-08-15 13:22:15 -04:00
Joseph Doherty 8da60f57ac feat(health): KPI history + sustained-backlog badge for LocalDb replication (row 33) 
LocalDbOplogBacklog was reported and Prometheus-exported with nothing consuming
it: not sampled into KPI history, so a purge-burst spike could not be reviewed
after the fact, and no threshold treatment anywhere. Both halves shipped.

KPI SAMPLING. SiteHealthKpiSampleSource now emits localDbOplogBacklog and
localDbReplicationConnected (0/1), and the backlog is charted on the Health
dashboard's per-site trend panel. Charting is the point: a point-in-time tile
can never answer 'did that spike drain?', which is the only thing separating an
expected purge burst from a stalled peer.

Both are CONDITIONALLY emitted rather than coerced with ?? 0 like the
auditBacklogPending metric beside them. Both health-report fields are nullable
and null means 'no reading' — replication not configured (rig site-b/site-c) or
a failed poll — never zero. A fabricated 0 would render a pair that cannot read
its own oplog, or one deliberately unreplicated, as perfectly healthy once a
minute forever, burying the real series. So a site emits 12 metrics when
unreplicated and 14 when replicated, and a GAP in these two series truthfully
means 'not replicated / not reporting'. Note false and null differ for the
connected flag: a configured link that is DOWN samples as 0; no link samples as
nothing. Both fold as Gauge (summing an oplog depth, or a 0/1 flag, per hour is
meaningless).

THRESHOLD SEMANTICS — sustained, not instantaneous. This is the design, not a
refinement. The site_events retention purge is CDC-captured like any other
write, so the backlog jumps by the deleted batch and drains as the peer acks
it; a bare 'backlog > threshold' badge would fire on every replicated site
every day and be ignored within a week. Healthy is defined by the backlog
RETURNING to ~0, which only a dwell can express. CentralHealthAggregator holds
LocalDbBacklogAboveSince — the instant the backlog first rose above
LocalDbBacklogThreshold (default 1000) — and the badge appears only after
LocalDbBacklogSustainedFor (default 15 min). A report at or below the threshold
clears the anchor; one above leaves the ORIGINAL instant intact, so the window
is sustained rather than sliding.

The null case is deliberately a third branch: a null backlog leaves the anchor
UNCHANGED rather than clearing it. Null is evidence neither of a drain nor of a
backlog, so a node going blind mid-climb must not silently reset its own dwell,
and an unreplicated site (null forever) never accumulates an anchor and can
never be badged.

Validation rejects a non-positive threshold (0 would badge every replicated site
permanently, since an idle backlog already sits at it) and a dwell shorter than
one ReportInterval (satisfiable by a single sample, which collapses the check
back into the bare threshold it replaces).

DOC CORRECTION worth flagging: this row's premise was 'mirror the SiteAuditBacklog
configurable threshold badge'. That badge does not exist — Component-HealthMonitoring.md
described it in two places but no option, no comparison and no markup were ever
implemented. The two stale claims are corrected here rather than left to send the
next reader looking for it; the LocalDb badge is built fresh on the Metrics-stale
badge's pattern, which is a real configurable-threshold badge.

Tests: HealthMonitoring 115 (9 new aggregator tests incl. purge-burst-drains,
unreplicated-never-anchors, null-mid-climb, exact-threshold, stale-report; 5 new
validator tests; 4 new sampler tests incl. the null-semantics and false-vs-null
cases); CentralUI 1042 (5 new badge/chart tests incl. the purge-burst negative
control); Commons Kpi 93; KpiHistory 54.
 2026-08-15 13:02:35 -04:00
Joseph Doherty 41a5ecf14a feat(external-systems): deploy per-system retry policy to sites + CLI/management surface (register row 27) 
An external system's MaxRetries/RetryDelay existed on the central entity but never
reached a site and had no management surface, so every ExternalSystem.CachedCall
buffered with the S&F engine default (DefaultMaxRetries 50 x DefaultRetryInterval
30s ~= 25 min to park) whatever central had configured. The sibling
DatabaseConnectionArtifact had always carried its equivalents.

The consumption end was already wired: ExternalSystemClient.CachedCallAsync already
read system.MaxRetries/RetryDelay and passed them to EnqueueAsync with the 0-as-unset
convention. Only the transport and persistence were missing.

Commons
- ExternalSystemArtifact gains MaxRetries/RetryDelay as optional positional params
  (0 / TimeSpan.Zero = unset), matching DatabaseConnectionArtifact rather than going
  nullable, so it maps 1:1 onto the entity and the proto3 scalars.
- Create/UpdateExternalSystemCommand gain nullable MaxRetries/RetryDelay, following
  the existing TimeoutSeconds "omit to preserve" convention. Both additive.

Central
- ManagementActor create maps both; update preserves-if-null.
- ArtifactDeploymentService projects them onto the artifact.

Wire
- site_command.proto ExternalSystemArtifactDto gains max_retries=7 /
  retry_delay=8 (google.protobuf.Duration), mirroring DatabaseConnectionArtifactDto.
  Additive: a pre-existing site decodes them as 0/absent, which IS the unset value,
  so old behaviour is preserved. Vendored C# regenerated via docker/regen-proto.sh
  sitecommand; csproj Protobuf item re-commented, no message types added or removed.

Site
- external_systems gains max_retries / retry_delay_ms via AddColumnIfMissing,
  DEFAULT 0 = unset. Storage upsert, repository SELECT/mapping (new ordinals 6/7
  appended so existing ones are untouched), apply path, and the legacy migrator's
  explicit column list all carry them.

Replicated-table decision: external_systems IS in SiteLocalDbSetup.ReplicatedTables,
so this was checked against the AckTime/native_alarm_state precedent. Real columns
are correct here, not a JSON bag: the table has no general-purpose config column
(auth_configuration and method_definitions are both semantically specific),
timeout_seconds already set the additive-column precedent on this exact table, and
sibling database_connections stores the same two values as plain columns. Verified in
LocalDb source that RegisterReplicated re-reads PRAGMA table_info every boot and its
install script is DROP-then-CREATE, so capture triggers rebuild over the new column
set automatically, and SiteStorageSchema.Apply runs before registration on the same
boot. The only cost is the ReplicatedTable.ComputeDigest change, which the sync
handshake compares fail-closed - a mixed-version pair refuses to sync loudly instead
of diverging silently. Both-nodes-together is already the standing site-pair rule, so
no new operational constraint; recorded in topology-guide.md.

Also fixed: ArtifactDiff.CompareExternalSystem never compared MaxRetries/RetryDelay,
so a bundle import that changed a system's retry budget previewed as "Identical" -
harmless while the values were inert, a live behaviour change now.

No EF migration: the central ExternalSystemDefinitions columns already exist
(InitialSchema). Central UI already had both form fields.

Tests: site storage round-trip + unset-default, site apply path, central artifact
projection, proto round-trip (via the shared sample), management create/preserve,
CLI option parse + alias, and diff reporting. Fail-before verified on the two core
tests (expected 7 got 0 on the read path; expected 9 got 0 on the apply path).

Docs: Component-ExternalSystemGateway, Component-StoreAndForward, Component-CLI,
components/StoreAndForward, components/SiteRuntime, deployment/topology-guide,
CLI README (which was also missing --timeout-seconds entirely).
 2026-08-15 12:56:12 -04:00
Joseph Doherty 0d2d6de6c6 chore(rig): set ASPNETCORE_ENVIRONMENT on the site containers (register row 63) 
The central containers have always set it; the six site containers never did,
so the Development-only DI scope validation from register row 50
(ServiceProviderValidation.Configure — ValidateScopes/ValidateOnBuild) was dead
on exactly the half of the topology that has six of the eight nodes. Program.cs
even carried a comment saying the site branch 'is the one that actually gains
coverage' while the rig gave it none; that comment is corrected here too.

Now that the site graph validates clean this buys live coverage: a captive
dependency or an unconstructible site-only descriptor fails the container at
boot instead of surfacing later as a mid-run resolve error.

Applied to all six docker/ site containers and both docker-env2/ site
containers, keeping the two rigs consistent. Dev/rig only — never production.

FILE EDITS ONLY: no docker command was run, and deploy.sh was not invoked.
Both compose files re-parsed with yaml.safe_load; all 8 + 4 services confirmed
to carry the expected env.
 2026-08-15 12:53:33 -04:00
Joseph Doherty 103f2f1931 feat(host+rig): give central a dedicated Prometheus scrape port (register row 56) 
The row read 'central nodes expose no Prometheus endpoint'. That is not quite
what was wrong, and the difference matters. Central already maps /metrics
(Program.cs central branch) and telemetry is registered for both roles through
the shared BindSharedOptions, so the endpoint rides endpoint routing and has
always been served on central's web port (:5000 on the rig, anonymous — there
is no FallbackPolicy). What central lacked was the site branch's dedicated
MetricsPort, which is the port the established live-scrape recipe probes:

    docker exec <node> bash -c 'exec 3<>/dev/tcp/localhost/8084 ...'

So a probe of central on 8084 failed and central-only series (KPI/outbox
counters, the bundle-import session gauge + its new rejection counter,
alarm-cache gauges) read as unscrapeable while they were in fact being
exported on a different port.

Central now binds a dedicated HTTP/1.1 listener on ScadaBridge:Node:MetricsPort,
so one probe recipe covers all eight rig nodes. HTTP/1.1-capable because a
standard Prometheus scraper cannot negotiate h2c, which is why this cannot
share the gRPC port.

Read as a PRESENCE check, not GetValue's defaulted read. NodeOptions defaults
MetricsPort to 8084, so a defaulted read would make every existing central node
start binding a new port on upgrade — and a port already in use is a hard boot
failure, not a degraded one. Central gains the listener only where an operator
asked for it; site nodes are unchanged and still bind theirs unconditionally.
The bind is also skipped when the port would collide with the gRPC port or an
ASPNETCORE_URLS port, since Kestrel throws on a duplicate ListenAnyIP and
/metrics is reachable on those ports anyway.

Config only — no host port mapping added, because the probe runs inside the
container. Applied to both docker/ centrals and both docker-env2/ centrals.

FILE EDITS ONLY: no docker command was run, and deploy.sh was not invoked.
 2026-08-15 12:53:26 -04:00
Joseph Doherty b5f120c370 feat(observability): honest stream-subscriber name + real data-connection gauges (row 54) 
scadabridge_site_connection_up never counted data connections. It counts open
central->site gRPC subscriber streams (Debug View + the site-wide alarm feed),
so it correctly reads 0 on a healthy site nobody is watching — and the name
invited reading it as device health. That misreading has already cost a live
debugging session: register row 48's 'connection_up is 0 on all six site
nodes' was healthy steady state, not the fault it was diagnosed as.

Two halves, per the row:

RENAME. scadabridge.site.connection.up -> scadabridge.site.stream.subscribers.active
(helpers SiteConnectionOpened/Closed -> SiteStreamSubscriberOpened/Closed), and
the instrument gains the unit:"1" the other gauges already carried. Every
reference updated: the gRPC server, both test suites, Component-Communication.md,
components/Commons.md, docker/README.md and docker/seed-live-probe.sh. A test
now pins the new name AND asserts the old one is gone, so it cannot creep back.

NEW. scadabridge.site.data_connections.connected / .total — the first exported
data-connection signal (the state existed on the health report and in the CLI,
but never as a metric). connected < total is the alert condition; neither
number alone separates 'a device is down' from 'nothing is configured', which
is why the denominator ships with the numerator.

Sourcing is deliberate on two counts. It is fed by the DCL at its four state
transitions rather than derived from the health report, so the metric survives
a dead report pipeline — the exact failure the 'metrics stale' flag exists to
catch, and a derived gauge would go blind at the same moment. And it uses a
static registry reached through ScadaBridgeTelemetry (the SiteConnectionOpened
precedent) rather than a DI-injected holder: the DCL actors already take
ISiteHealthCollector through their constructors, and threading a second
dependency through both would ripple into every test that builds them. It also
keeps SiteHealthCollector untouched, which matters right now — another agent is
concurrently editing that file for the tag-resolution path.

Removal is wired too: an undeployed connection leaves the registry, so it stops
counting toward the total instead of latching connected < total on forever.

Both gauges read 0 on the standby (connections open on the active node only) —
healthy, documented, and called out in the docs so it does not become the next
site_connection_up.

No ObservedMeters allowlist change: all three instruments ride the existing
ZB.MOM.WW.ScadaBridge meter. (An unlisted meter exports nothing, silently.)

Tests: Commons 711 passed (9 new, delta-based against a baseline with unique
per-test connection names, so they do not inherit the register row 32
QueueDepthGaugeTests static-state flake); Communication 717; DCL 319.
 2026-08-15 12:53:12 -04:00
Joseph Doherty abf864b4e5 docs(siteruntime): record reconcile's runtime heal semantics (register row 67) 
Component-SiteRuntime.md gains a Startup Behavior note explaining why storing a
config started nothing (the reconcile pass fires after the boot-time SQLite load,
and Instance Actors are created only by that load or an explicit deploy/enable)
and what the EnsureInstanceStartedCommand path now guarantees: singleton-proxy
routing, idempotence, durable-row authority over enabled/disabled, deferral to
in-flight deploys, quiet no-op on an unreplicated peer's heal, and the Warning
log + `deployment` site event that make the heal visible. The command is also
listed under Instance Lifecycle Commands, flagged site-internal and reply-less so
it is not mistaken for a central-sent command.

docs/components/SiteRuntime.md gets the matching implementation-level note beside
the surviving-reconcile paragraph in Standby replication.
 2026-08-15 12:50:24 -04:00
Joseph Doherty 5da27e05c7 fix(dcl): push tag-resolution counts on first subscribe (register row 55) 
A site health report showed tagResolutionCounts 0/0 for a connection whose
dataConnectionTagQuality on the SAME report read good:2 — reproduced live on
the docker rig against site-a's rig-probe-1 instance (2 Good OPC UA tags on
"OPC PLC Simulator"), on the ACTIVE node's own report.

Not a standby-shadowing or keying/removal problem, and not a serialization
problem: both dictionaries live on one collector instance on one node, and the
CentralControlDtoMapper round-trip is correct in both directions. The actor's
own derived counts (TotalSubscribedTagCount / ResolvedTagCount) were right the
whole time — GetHealthReport replied 2/2.

Root cause: HandleSubscribeCompleted — the path that registers tags on a first,
healthy subscribe — mutated _instancesByTag and _subscriptionIds but never
pushed to ISiteHealthCollector. The only pushes were BecomeConnected's (fires
before any instance subscribes, so its 0/0 is legitimate), the batch apply
(reconnect re-subscribe / resolution probe) and unsubscribe (teardown). A
connection whose tags all resolve first time reaches none of those, and
ScheduleTagResolutionRetry cancels the retry timer outright when nothing is
unresolved — so the connect-time 0/0 was never superseded and stood forever.

The blind spot was therefore worst on a perfectly healthy connection, which is
exactly when the "device still booting / tag path unresolved" diagnostic should
read clean rather than empty. Any reconnect masked it (the batch path pushes),
which is why it presented only on a clean first deploy.

Fix: push the derived counts at the end of the first-subscribe apply, mirroring
the existing push site in HandleBatchSubscribeCompleted. No semantic change to
the null/no-data convention — the map entry already existed, its values were
simply stale zeros.

Tests: fail-before regression test pins the exact mechanism (NSubstitute records
a single UpdateTagResolution(name, 0, 0) pre-fix while the actor reports 2/2).
DCL 320, HealthMonitoring 97, SiteRuntime 605, Communication 717 all green;
slnx builds clean.

Docs: Component-DataConnectionLayer.md said the counts were "derived at report
time", which implies a pull that cannot go stale and papered over this bug.
Corrected to state they are pushed, name the four push sites, and record why a
healthy connection was the worst case.
 2026-08-15 12:46:37 -04:00
Joseph Doherty 65b19516c8 test(storeandforward): isolate the process-global queue-depth gauge — register row 32 
QueueDepthGaugeTests failed intermittently whenever a sibling test class started a
StoreAndForwardService concurrently. `scadabridge.store_and_forward.queue.depth` is an
ObservableGauge on ScadaBridgeTelemetry's static meter reading ONE process-global
provider slot; every StartAsync writes it, last registrant wins. A concurrent StartAsync
clobbered the gauge mid-assertion and the test read the sibling's depth (observed:
expected 0, actual 6 — 2 runs in 5 with parallel collections enabled).

The global slot cannot be swapped for a per-test meter: three of the six facts assert
the slot-ownership semantics themselves (compare-and-clear identity, stop/restart
re-seed), so the slot IS their subject. The correct isolation is mutual exclusion —
QueueDepthGaugeTests, StoreAndForwardServiceTests and CachedCallAttemptEmissionTests
(the three classes that call StartAsync) now share the QueueDepthGaugeCollection, which
xUnit runs serially; the other eight classes stay parallel.

With real isolation in place the assembly-wide `parallelizeTestCollections: false`
blanket switch is no longer needed and is flipped back on, so the isolation is actually
exercised rather than papered over.

QueueDepthGaugeCollectionGuardTests is a staleness self-test: it scans this project's
sources (anchored by [CallerFilePath], not the output dir) for files that both construct
and start a service, and fails if one omits the collection attribute — so a new test
class cannot silently reintroduce the flake. Negative-controlled: dropping the attribute
from StoreAndForwardServiceTests makes the guard fail with the offending file name, and
the original gauge flake reappears in the same run.

Verification: 8 consecutive full-suite runs green with parallel collections ON (135
tests), plus 5 more after the negative control was reverted. Test-only change; no src/
involvement.
 2026-08-15 12:44:14 -04:00
Joseph Doherty 482165cbac fix(siteruntime): startup reconcile starts the healed InstanceActor (register row 67) 
SiteReconciliationActor's node-STARTUP fetch healed the config FILE but never the
RUNTIME. Instance Actors are created from exactly two paths -- the Deployment
Manager's boot-time SQLite load and an explicit deploy/enable command -- and the
reconcile pass runs AFTER that boot load has already read storage. So a config
reconcile restored started no actor: an enabled, successfully-deployed instance
whose config was missing at boot (lost volume, first boot after a restore,
rehomed data dir) stayed dead until the NEXT restart, whose load finally picked
it up. Proven live on site-a 2026-08-15: "Reconcile: fetched + stored config"
with no "InstanceActor started" after it, and a debug snapshot answering
instanceNotFound.

The pass now raises a fire-and-forget EnsureInstanceStartedCommand (new Commons
Deployment message) for every ENABLED instance it healed, routed through the
deployment-manager singleton PROXY so Instance Actors stay under the singleton no
matter which node's reconcile found the gap.

The handler is deliberately advisory:
- idempotent -- no-op when the actor is already running, which both nodes'
  reconcile actors legitimately trigger for the same instance;
- enabled-state-aware -- it re-reads the durable row, which is the authority on
  both the config body and enabled/disabled, so a disable landing after the
  fetch still wins and the instance stays stopped;
- it defers to any in-flight deploy/redeploy/warm for the instance, which keeps
  the deploy gate's ordering guarantees and the all-or-nothing-per-instance rule
  intact -- this never applies a config, it only starts one already persisted;
- a missing row is a quiet no-op, the normal case on an unreplicated site whose
  peer node healed its own file.

The heal is no longer silent: a successful reconcile start emits a Warning log
line and an operational `deployment` site event, because reaching it means the
node booted without a config for an instance that should have been running.
 2026-08-15 12:43:25 -04:00
Joseph Doherty 90ce099928 feat(transport): counter for import-session cap rejections (register row 66) 
Row 53 shipped the scadabridge.central.transport.import_sessions.open gauge
and a Warning log on a cap hit, but a log line is not something an operator
can alert on. Adds the companion counter

    scadabridge.central.transport.import_sessions.rejected

on the existing ZB.MOM.WW.ScadaBridge meter, so no ObservedMeters allowlist
change is needed (an unlisted meter exports nothing, silently).

The emit point is deliberately narrow. BundleSessionStore.Open tests the cap
TWICE — once on entry and again after EvictExpired() — and only the second
test is a rejection: a store full of merely expired sessions reclaims a slot
and admits the caller. Counting the first would make the series fire
routinely on any node whose sweep had not yet run, turning a page-worthy
signal into background noise. Re-Opening an already-open session id skips
the cap block entirely and is likewise not counted.

The two instruments are diagnostic as a pair: rejections rising while the
gauge oscillates below the cap is genuine concurrency (raise the cap);
rejections rising while the gauge sits pinned at the cap is a session leak
(register row 43's signature), where raising the cap only defers the problem.

Tests assert a delta over a scoped action rather than an absolute value, so
they do not inherit the register row 32 static-state flake that makes
QueueDepthGaugeTests fail only under the full concurrent suite. Negative
controls cover under-cap, expired-reclaim, and duplicate-session-id.

Tests: 11 passed in BundleSessionObservabilityTests (6 existing + 5 new).
 2026-08-15 12:42:56 -04:00
Joseph Doherty ef876b38c5 chore(docker+register): refuse worktree deploys (relative bind mounts rehome rig data — register row 68); record row 67 (startup reconcile stores config but starts no InstanceActor) + row-48 postscript 2026-08-15 12:18:01 -04:00
Joseph Doherty 002299c400 docs(register): close rows 46, 48-53 with evidence; add rows 54-66 from the fix wave's discoveries 2026-08-15 12:03:21 -04:00
Joseph Doherty cde4af77fd merge: audit-config query fix + rig live-probe seed (register rows 46, 48) — ValueTuple serialized as {} fixed with AuditLogQueryResult record + ITuple guard; docker/seed-live-probe.sh 2026-08-15 12:00:32 -04:00
Joseph Doherty 4785f3cbef merge: transport hardening (register rows 51-53) — BundleLockedException curated to Commons/MapFault, fleet-wide MS SQL unlock-throttle window, import-session gauge + cap Warning 2026-08-15 12:00:28 -04:00
Joseph Doherty 1be08e96cc merge: code-review scoped-lifetime sweep (register row 49) — no live bug; NotificationOutbox-004 resolution note recorded 2026-08-15 12:00:28 -04:00
Joseph Doherty a404854212 merge: captive-dependency guard (register row 50) — scope validation in Development + actor field scan; fixes two Site-role unconstructible registrations 2026-08-15 12:00:28 -04:00
Joseph Doherty 73f518d6a6 chore(transport): delete the dead BundleUnlockRateLimitedException 
Superseded by BundleUnlockThrottledException when register row 44 wired the
limiter live and moved the typed exception to Commons so ManagementService's
MapFault could curate it. The original type has been unreferenced since —
never thrown, never caught, only its own definition — and with row 51 moving
BundleLockedException into the same Commons namespace there are now three
similarly-named unlock exceptions, one of which does nothing. A future
maintainer catching the dead one would silently get no coverage.

Solution builds clean with it gone (zero references). Transport 173 passed.
 2026-08-15 11:59:06 -04:00
Joseph Doherty 00a804e6d9 guard(di): captive-dependency tripwires — scope validation + an actor field scan 
Register row 50. Row 45 (fixed 9e769735) was a live captive dependency —
NotificationOutboxActor cached delivery adapters resolved from an actor-lifetime
IServiceScope, so MaxParallelDeliveries concurrent deliveries shared one EF
DbContext and central-b logged EF's "second operation was started on this context
instance" continuously. Actors take the ROOT IServiceProvider rather than typed
services, which is invisible to build-time analysis, so the bug class had no
mechanical guard at all. This adds both of the register's mitigations; neither
subsumes the other.

1. ValidateScopes + ValidateOnBuild in Development, on BOTH branches, via
   ServiceProviderValidation.Configure. WebApplication.CreateBuilder already sets
   both flags to IsDevelopment(), so on paper this restates a default — it is
   written out because that default is silent (a future move off CreateBuilder
   drops it with no diagnostic) and because this host carries three
   environment-ish signals (SCADABRIDGE_CONFIG, DOTNET_ENVIRONMENT,
   ASPNETCORE_ENVIRONMENT), so "is this node in Development" is not obvious at the
   call site. Production stays unvalidated deliberately: a validation failure is
   fail-closed, and CI now proves what a production node will not check.

   This surfaced two REAL latent violations on the SITE role — not captive
   dependencies but unconstructible descriptors, which fail ValidateOnBuild for
   the entire graph and are therefore what kept the site branch from being
   validatable at all (SiteHealthEndpointTests had pinned UseEnvironment
   ("Production") to work around exactly this):

     - ReconcileService (scoped, needs the central-only IDeploymentManagerRepository)
       moved out of the shared AddCommunication() into a new AddCentralCommunication().
     - AuditLogKpiSampleSource (scoped IKpiSampleSource, needs the central-only
       IAuditLogRepository) moved out of the shared AddAuditLog() into a new
       AddAuditLogCentralKpi().

   Both were registered on sites under the argument "harmless — only the central
   actor ever resolves it". True of the runtime, false of the container: a
   registration nothing can build is what stops the validator being usable, and it
   masks whatever real defect the validator would have found. Both are now
   central-only and SiteHealthEndpointTests boots the site branch validated again.

2. CaptiveDependencyGuard: a reflection scan over every actor class in the shipped
   assemblies (28 today; the reference graph is walked from the Host assembly
   because referenced assemblies load lazily). Fails on any instance field — auto-
   property backing fields included, they are ordinary private fields — typed as
   IServiceScope, as a DbContext, or as a service the REAL composition roots
   register Scoped. The scoped set is derived from the built Central and Site
   containers rather than hand-listed, so a lifetime change automatically re-scopes
   the check. IServiceProvider / IServiceScopeFactory fields are explicitly allowed:
   that IS the correct per-operation-scope pattern.

   No live violation exists — every actor opens a scope per operation. Two fields
   are allowlisted with their safety argument rather than exempted silently:
   AuditLogIngestActor._injectedRepository and SiteCallAuditActor._injectedRepository
   are scoped repository types assigned ONLY by explicitly test-mode constructors,
   null under production wiring (AkkaHostedService uses the IServiceProvider
   overload in both cases), and SiteCallAuditActor additionally wraps its injected
   instance in a SerializedRepository. The shape is one constructor call away from
   being row 45 again, so it is recorded, not dismissed.

Tests: 10 new. Six permanent self-tests of the guard, including positive controls
that reproduce the pre-row-45 shape (an actor caching a scoped service, an
IServiceScope, a DbContext, and a scoped auto-property) and a negative control for
the allowed root-provider pattern — a reflection guard that matches nothing passes
just as green as one that works. Plus an actor-count floor with cross-assembly
spot-checks, an allowlist-staleness test that fails any entry no longer matching a
flagged field, and the two per-role scans. Verified non-vacuous by emptying the
allowlist: the Central scan then names both real fields. Two composition-root
build tests force both validation flags on per role regardless of environment.

Doc: REQ-HOST-11 in Component-Host.md records all three rules and their guards.
 2026-08-15 11:58:40 -04:00
Joseph Doherty 677263af57 feat(transport): fleet-wide bundle-unlock throttle window (register row 52) 
The per-IP unlock throttle's counters were a process-local ConcurrentDictionary
on each central node, so the effective fleet cap was N x
MaxUnlockAttemptsPerIpPerHour per client (failing over to the other node handed
the client a fresh budget) and a node restart cleared the window outright.

The window now lives in the central MS SQL BundleUnlockAttempt table — the HA
store both central nodes already share, so no new infrastructure, no new port,
no consensus protocol:

- Commons: BundleUnlockAttempt entity + IBundleUnlockAttemptRepository.
- ConfigurationDatabase: entity config (one index,
  IX_BundleUnlockAttempt_Client_Attempted, serving both the windowed count and
  the prune), repository (AsNoTracking count, ExecuteDeleteAsync prune),
  DbSet, DI registration, migration AddBundleUnlockAttemptTable + the
  hand-written idempotent production script under docs/plans/sql/.
- Transport: IBundleUnlockAttemptWindow seam + SqlBundleUnlockAttemptWindow
  (singleton; resolves the scoped repository from its own scope per call, like
  the central purge singletons). BundleUnlockRateLimiter keeps the policy and
  gains TryRegisterAttemptAsync; BundleImporter.LoadAsync calls it.

Behaviour preserved from row 44: locked bundles spend no hourly budget (the
lockout check still runs first), throttle denials burn no per-bundle strike and
are NOT recorded so they cannot roll the window forward, unattributable clients
share the stricter <unattributed> bucket, and the budget counts attempts rather
than failures. Config keys and their semantics are unchanged, so the rig's 500
override stays valid.

Three deliberate trades, documented in both Transport docs:
- FAILS OPEN. A store error logs a Warning naming the client and cap, and
  permits the attempt. This is a best-effort rate cap over an already
  authenticated, Administrator-gated surface — the availability of legitimate
  imports outranks the cap, and a DB blip must not make the import wizard
  unusable. Falling back to the process-local window was considered and
  rejected: it would still block a legitimate operator's 11th import.
- Approximate. Count-then-insert is not atomic across nodes, so simultaneous
  attempts can overshoot by the number in flight — bounded, self-correcting
  within the window, and not worth provider-specific raw SQL.
- Pruned opportunistically on the unlock path (CAS-claimed at most every 15
  min per process, bounded to 1000 rows, retaining 2x the window, best-effort
  try/catch) rather than by a new background service — Transport owns no
  central scheduler. Same shape as DeploymentService's terminal-record sweep.

Tests: Transport.IntegrationTests 116 passed (+10 — two limiter instances over
one store enforcing a single combined window, restart durability, denials
burning no budget, sliding release, bucket isolation, store-failure fail-open,
caller-cancellation not swallowed, no-shared-window fallback, prune and its
negative control); ConfigurationDatabase 395 passed (+5).
 2026-08-15 11:52:34 -04:00
Joseph Doherty b5e6821caa chore(docker): seed script for a deployable live-data probe instance on the rig 
Deferred-work register row 48. The rig had no deployable instance — all seven
soak-* rows fail pre-deployment validation (their bound data connections no longer
exist on the sites) — so debug snapshot returned instanceNotFound, no gRPC site
stream was ever opened, and the R2 batching histogram
scadabridge.site.stream.batch_size had nothing to record. Any live probe of the
data or streaming path was unverifiable.

seed-live-probe.sh rebuilds the minimum working set via the CLI only (no DB edits):
RigProbe template (Spike <- ns=3;s=SpikeData, RandomInt <- ns=3;s=RandomSignedInt32,
both ~1 Hz; Heartbeat static attribute + a trust-clean 5s interval script) and a
deployed rig-probe-1 on site-a against connection 1.

Records the two traps that cost time here:
- The custom tags in infra/opcua/nodes.json (Motor.*, Tank.*, Valve.*,
  TestChildObject.*) are DECLARED but not simulated by opc-plc — they read 0/false/
  epoch forever, so binding to them yields a Connected, Good-quality connection with
  zero event traffic. Only the built-in nodes move.
- scadabridge_site_connection_up counts open central->site gRPC SUBSCRIBER STREAMS,
  not OPC UA data connections. It reads 0 on a healthy site with no subscriber, which
  is how row 48 read it as a data-connection fault. Data-connection health lives in
  the health report's dataConnectionEndpoints / dataConnectionTagQuality.
 2026-08-15 11:46:57 -04:00
Joseph Doherty 75520beece docs(code-reviews): record the missing NotificationOutbox-004 resolution note 
Deferred register row 49 (scoped-lifetime sweep). NotificationOutbox-004 was
flipped to Status: Resolved but its Resolution body still read _Unresolved._,
so the file claimed an open DI-scope-lifetime hazard that the code had already
closed. Re-verified against current code: EmitAttemptAuditAsync /
EmitTerminalAuditAsync now await _auditWriter.WriteAsync inside the existing
try/catch (NotificationOutboxActor.cs:731, :785) — the preferred remedy from
the finding's own Recommendation — so the audit task cannot outlive the
per-sweep DI scope, the catch is reachable for writer faults, and the
misleading 'writer never throws' XML-doc is gone.

Documentation only; no code touched. Solution build clean (0 warnings,
0 errors).
 2026-08-15 11:42:55 -04:00
Joseph Doherty 372ea4df1b feat(transport): observability for the import-session cap (register row 53) 
Nothing counted open import sessions and nothing logged a rejection at
MaxConcurrentImportSessions (default 8), so the register-row-43 session leak
could only be inferred from rotating test failures.

- Gauge scadabridge.central.transport.import_sessions.open on the existing
  ZB.MOM.WW.ScadaBridge meter — already in SiteServiceRegistration.ObservedMeters
  (the ZbTelemetryOptions.Meters allowlist), so no telemetry config change is
  needed and the series is not silently dropped. Provider-backed, mirroring the
  StoreAndForward queue-depth gauge: registered by BundleSessionEvictionService
  .StartAsync (Transport's only lifecycle seam) and identity-checked-cleared on
  StopAsync so a late stop cannot clobber a newer registrant.
- IBundleSessionStore.OpenSessionCount counts only UNEXPIRED sessions. The raw
  dictionary count would report the node as pinned at the cap for up to a
  one-minute sweep interval after a burst had drained — the exact false positive
  the instrument exists to rule out.
- BundleSessionStore.Open logs Warning with open count, cap and TTL when it
  rejects. Fires only on a real cap hit: a store full of expired sessions is
  reclaimed first and logs nothing.
- Docs: docs/components/Transport.md + docs/requirements/Component-Transport.md.

Tests: Transport 173 passed (+6 new — Warning content, two negative controls,
unexpired-only counting, gauge register/clear, no-clobber-on-late-stop).
 2026-08-15 11:42:31 -04:00
Joseph Doherty 09108018f1 fix(management): audit-config query returned {} for every filter — project the repo ValueTuple into a named result record 
Deferred-work register row 46. HandleQueryAuditLog returned ICentralUiRepository's
(IReadOnlyList<AuditLogEntry> Entries, int TotalCount) ValueTuple straight to
ManagementActor.SerializeResult. System.Text.Json writes properties only, and a
ValueTuple exposes Item1/Item2 as public FIELDS, so the whole payload serialized to
the literal "{}" — the CLI faithfully printed an empty object no matter how many
AuditLogEntries rows matched, with no error anywhere in the stack.

Fixed at the handler (the layer that leaks the tuple), not the CLI: new Commons record
AuditLogQueryResult(Entries, TotalCount), mirroring the SecuredWriteListResult shape
already used by the other paged management query.

Also guards the whole class of bug: SerializeResult now throws on any ITuple result,
so the next handler that returns a tuple fails loudly instead of silently emitting {}.

Tests (fail-before verified — the query test asserted exactly "{}"):
- QueryAuditLogCommand_ReturnsEntriesAndTotalCount
- SerializeResult_TupleResult_ThrowsInsteadOfEmittingEmptyObject
- SerializeResult_AuditLogQueryResult_EmitsEntriesAndTotalCount

Docs: Component-CLI.md + CLI README document the response shape and why it is a record.
 2026-08-15 11:38:28 -04:00
Joseph Doherty b449fe201a fix(transport): curate BundleLockedException on the management surface (register row 51) 
ManagementService references Commons but not Transport, so MapFault could not
see BundleLockedException and the per-bundle 3-strike lockout reached the CLI
as "An internal error occurred (CorrelationId=...)" — the operator got a
correlation id instead of the reason, and no hint that further attempts on
that bundle would be refused.

Applies the pattern register row 44 established for BundleUnlockThrottledException:

- Relocate BundleLockedException from Transport/Import/ to
  Commons/Types/Transport/, beside BundleUnlockThrottledException. Both
  consumers (BundleImporter, CentralUI TransportImport) already carried the
  Commons.Types.Transport using, so no call-site edits were needed.
- Add curated catch arms in ManagementActor.HandlePreviewBundle and
  HandleImportBundle that rethrow as ManagementCommandException, so the
  message survives MapFault's disclosure filter.
- Mirror the row-44 throttle fault-mapping test for both handlers.
- Document the relocation in docs/components/Transport.md and
  docs/requirements/Component-Transport.md.

Tests: ManagementService 499 passed (+2 new), Transport 167 passed.
 2026-08-15 11:37:36 -04:00
205 changed files with 23284 additions and 1380 deletions
Expand all files Collapse all files
CLAUDE.md
+4 -4
View File
@@ -83,7 +83,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
- `site_events.id` changed from autoincrement INTEGER to an application-minted **GUID**. Last-writer-wins keys on the primary key, so two nodes independently minting `id=1,2,3…` would destroy each other's events rather than merge them. The event-log read path uses a composite `(timestamp, id)` keyset cursor with an **opaque string** continuation token; `EventLogEntry.Id` and both `ContinuationToken`s are `string`/`string?` on the site↔central Akka DTOs.
- `ScadaBridge:OperationTracking:ConnectionString`, `ScadaBridge:SiteEventLog:DatabasePath` and — as of Phase 2 — `ScadaBridge:StoreAndForward:SqliteDbPath` + `ScadaBridge:Database:SiteDbPath` are all **migration-only** — nothing reads them but `SiteLocalDbLegacyMigrator`, which copies a pre-Phase-1 file in once (deterministic `mig-{NodeName}-{legacyId}` event ids, `INSERT OR IGNORE`, runs AFTER `RegisterReplicated` so migrated rows replicate) and renames it `.migrated`. Delete the keys once a node has migrated.
- This incidentally fixes a data-loss bug: both legacy databases defaulted to CWD-relative paths **outside** the mounted volume and were discarded on every container recreate.
- **Replication is default-OFF and opt-in** via `LocalDb:Replication:PeerAddress` + a matching `ApiKey` on both nodes. `LocalDbSyncAuthInterceptor` is **fail-closed**: no configured key means no sync stream is accepted at all, so a key typo does not degrade to unauthenticated replication — the pair simply stops converging. The sync endpoint shares the existing site gRPC h2c listener (8083); no new port. Rig posture: **site-a replicated, site-b/site-c deliberately not**, so both states are proven side-by-side. Status surfaces on the site health report as `LocalDbReplicationConnected` / `LocalDbOplogBacklog` (both nullable — null means "no data", NOT "disconnected with an empty backlog") and as `localdb_*` Prometheus series. **A transient backlog spike is expected daily and is not a fault:** the `site_events` retention purge (and the storage-cap trim) is CDC-captured like any other write — there is deliberately no purge-exemption path — so the backlog jumps by the deleted batch size and drains as the peer acks it; the purge logs an Information note saying so on a replication-enabled node (`SiteEventLogReplicationCheck`, supplied by the Host from `SiteLocalDbSetup.ReplicationIsConfigured` so SiteEventLogging never reads LocalDb config itself). Healthy = `LocalDbReplicationConnected` stays true and the backlog returns to ~0; see `docs/deployment/topology-guide.md` → *Reading the replication backlog*. Note `ZbTelemetryOptions.Meters` is an **allowlist** (`SiteServiceRegistration.ObservedMeters`); an unlisted meter exports nothing, silently.
- **Replication is default-OFF and opt-in** via `LocalDb:Replication:PeerAddress` + a matching `ApiKey` on both nodes. `LocalDbSyncAuthInterceptor` is **fail-closed**: no configured key means no sync stream is accepted at all, so a key typo does not degrade to unauthenticated replication — the pair simply stops converging. The sync endpoint shares the existing site gRPC h2c listener (8083); no new port. Rig posture: **site-a replicated, site-b/site-c deliberately not**, so both states are proven side-by-side. Status surfaces on the site health report as `LocalDbReplicationConnected` / `LocalDbOplogBacklog` (both nullable — null means "no data", NOT "disconnected with an empty backlog") and as `localdb_*` Prometheus series. **An UNCONFIGURED node reports null for both** — the code only started matching that documented contract with register row 80: the engine is registered unconditionally and answers `Connected=false`/`0` while inert, and publishing that made site-b/site-c look like broken replicated sites and starved central's null branches (row 33's conditional KPI emission + sustained-badge anchor exclusion) of the null they exist for. `LocalDbReplicationStatusReporter` now gates on a Host-supplied `SiteLocalDbSetup.ReplicationIsConfigured` flag, so `false` means exactly one thing: **configured, peer session down** = a real fault. Watch the trap that caused it — configured-ness is `PeerAddress` **OR** `ApiKey`, so "no PeerAddress" is NOT "unreplicated" (the pre-existing Host wiring test fixture is a passive-half replicated node for precisely this reason). **A transient backlog spike is expected daily and is not a fault:** the `site_events` retention purge (and the storage-cap trim) is CDC-captured like any other write — there is deliberately no purge-exemption path — so the backlog jumps by the deleted batch size and drains as the peer acks it; the purge logs an Information note saying so on a replication-enabled node (`SiteEventLogReplicationCheck`, supplied by the Host from `SiteLocalDbSetup.ReplicationIsConfigured` so SiteEventLogging never reads LocalDb config itself). Healthy = `LocalDbReplicationConnected` stays true and the backlog returns to ~0; see `docs/deployment/topology-guide.md` → *Reading the replication backlog*. **That spike is PACED as of row 34 (2026-08-15)** — `ScadaBridge:SiteEventLog:MaxPurgedRowsPerTick` (default **250 000**, `0` = unlimited escape hatch) is a per-tick delete ceiling **shared** by the retention purge and the cap trim, and a ceiling-limited tick reschedules after `PurgeCatchUpInterval` (default **5 min**) instead of the daily `PurgeInterval` so retention still converges (~11 ticks ≈ 55 min for the worst case, vs 11 days without the acceleration). Measured worst case, unpaced: a 1 GB event log holds **~2.73 M rows at 393 B/row**, and expiring all of them in one tick took 285 s and queued **2 731 947 oplog rows — 2.7x LocalDb's `MaxOplogRows` default of 1 000 000**, i.e. enough to make the maintenance tick prune the oplog and flag a **full ten-table snapshot resync**. **The subtlety that makes pacing the right fix rather than a bigger `MaxOplogRows`:** the *resident* backlog is `produced − drained`, so a healthy peer blunts it — on a loopback pair the same 2.73 M burst peaked at only **391 k** and drained in 6.0 min. The overrun is therefore **conditional on peer throughput** (breached outright with the peer down; breached at ~1.5 M if the peer runs at half loopback speed), and a 250 k ceiling makes the outcome independent of the peer instead. Drain speed itself is not a bottleneck (7.6–19.8 k tombstones/s measured), so a full 250 k tick clears far inside row 33's 15-minute sustained dwell — verified at 2x a tick (500 k drained in 46 s). The purge loop is now a delay loop rather than a `PeriodicTimer`, so its cadence anchors to the END of the previous purge. Ceiling hits log a rate-limited (15 min) Warning and always increment `scadabridge.site.event_log.purge_ceiling_hits` — read the counter's shape: a run that stops is a backlog converging, hits that never stop mean the ceiling is mis-sized for that site's event rate. Measurement + decision record: `docs/plans/2026-08-15-site-events-purge-burst-measurement.md`. Note `ZbTelemetryOptions.Meters` is an **allowlist** (`SiteServiceRegistration.ObservedMeters`); an unlisted meter exports nothing, silently.
- Design: **scadaproj** `docs/plans/2026-07-19-scadabridge-localdb-design.md` (that doc lives in the umbrella repo, not here); Phase 1 plan + Phase 2 gate are here under `docs/plans/`. Phase 2 **deleted** `SiteReplicationActor`, its `ReplicationMessages`, StoreAndForward's `ReplicationService`, and `StoreAndForwardStorage.ReplaceAllAsync` — do not reintroduce them:
- CDC replication does all three jobs now: config deploys reach the standby as ordinary row changes — **the standby makes no fetch at all** during a deploy (`SiteReconciliationActor`'s node-STARTUP fetch when central reports gaps is a different, surviving path) — and buffer mutations replicate via triggers on `sf_messages`. `ReplaceAllAsync` was a destructive delete-all-then-insert-all resync and is **unsafe to reintroduce**: a mass DELETE on a replicated table would be captured and shipped to the peer. LocalDb's snapshot resync merges per row under LWW and never deletes, which is also why the old N1 directional-authority guard is gone — there is no wipe left to gate.
@@ -175,12 +175,12 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
- Two-person MxGateway secured writes (M7): two new global roles — `Operator` (initiates) + `Verifier` (approves) — added alongside the canonical `Administrator`/`Designer`/`Deployer`/`Viewer`, with `RequireOperator`/`RequireVerifier` policies. An Operator submits a secured write from the Central UI Secured Writes page (`/operations/secured-writes`); it stays a `Pending` `PendingSecuredWrite` row until a *distinct* Verifier approves it (no-self-approval enforced server-side in the ManagementActor, plus a compare-and-swap race guard). Approval relays a `WriteTagRequest` to the site MxGateway; MxGateway-protocol connections only; each lifecycle event (submit/approve/reject/execute) emits a best-effort `AuditChannel.SecuredWrite` / `AuditKind.SecuredWrite*` central-direct-write row sharing the row id as `CorrelationId`. (SecuredWrite audit rows stamp `SourceNode` via `ICentralAuditWriter`/`INodeIdentityProvider`.) Pending secured writes expire server-side after a configurable TTL (`ManagementServiceOptions.SecuredWritePendingTtl`, default 24 h): an overdue `Pending` row is CAS'd to `Expired` (never relayed) — enforced at approve/reject and swept opportunistically on list (arch-review S2, `AuditKind.SecuredWriteExpire`).
### Cluster & Failover
- **`auto-down` downing strategy (decision 2026-07-21 — availability over partition-safety).** Akka's `AutoDowning` provider, `auto-down-unreachable-after` = 15s: the leader among the REACHABLE members downs the unreachable peer, so a hard crash of EITHER node (active/oldest included) fails over to the survivor in ~25s. Accepted trade: a real partition → dual-active until an operator restarts one side. `keep-oldest` remains a supported `SplitBrainResolverStrategy` value (partition-safe, but an oldest-crash is a total outage — Akka's `down-if-alone` only rescues a side with ≥2 members, proven live + in 1.5.62 source). Decision record: `docs/plans/2026-07-21-auto-down-availability-decision.md`.
- **`auto-down` downing strategy (decision 2026-07-21 — availability over partition-safety).** Akka's `AutoDowning` provider, `auto-down-unreachable-after` = 15s: the leader among the REACHABLE members downs the unreachable peer, so a hard crash of EITHER node (active/oldest included) fails over to the survivor in ~25s. Accepted trade: a real partition → dual-active until an operator restarts one side. `keep-oldest` remains a supported `SplitBrainResolverStrategy` value (partition-safe, but an oldest-crash is a total outage — Akka's `down-if-alone` only rescues a side with ≥2 members, proven live + in 1.5.62 source). Decision record: `docs/plans/2026-07-21-auto-down-availability-decision.md`. **Ignore the boot-time Akka warning claiming "keep-majority will be used instead"** — drill-verified benign (2026-08-15, decision-doc postscript): the substitution does not happen with the explicit `AutoDowning` provider-class `BuildHocon` sets. But `AutoDowning` IS deprecated, so **every Akka.NET version bump must re-run `DRILL_MODE=active bash docker/failover-drill.sh`** (register row 70).
- Both nodes are seed nodes. `min-nr-of-members = 1`.
- Failure detection: 2s heartbeat, 10s threshold. Total failover ~25s (drill-measured 2026-07-21 under auto-down: active-crash TAKEOVER in 28s, standby-crash removal in 27s with 0 routing blips — `docker/failover-drill.sh`).
- CoordinatedShutdown for graceful singleton handover.
- Automatic dual-node recovery from persistent storage.
- **Active/standby is decided by `ActiveNodeEvaluator.SelfIsOldestUp`, never by cluster leadership** — see the Architecture note above. `/health/active` is **central-only** (site nodes map no `/health/*` at all) and backs both Traefik's active-node routing and `IActiveNodeGate`, so the proxy and the Inbound API always agree on which node is active. Central reaches a site by dialling `GrpcNodeAAddress`/`GrpcNodeBAddress` explicitly and flipping on error; whichever node answers, the site-internal `ClusterSingletonProxy` lands the work on the active node for free, so central still never needs to track which *site* node is active.
- **Active/standby is decided by `ActiveNodeEvaluator.SelfIsOldestUp`, never by cluster leadership** — see the Architecture note above. **Central's** `/health/active` backs both Traefik's active-node routing and `IActiveNodeGate`, so the proxy and the Inbound API always agree on which node is active. **Site nodes DO map `/health/*` too** — the "site nodes map no `/health/*` at all" claim that stood here was stale: `SiteServiceRegistration` registers `akka-cluster` + `localdb` (Ready) and `SitePairActiveNodeHealthCheck` (Active, role-scoped to `site-{SiteId}`, deliberately NOT central's unscoped shared check), and `Program.cs`'s Site branch calls `MapZbHealth()` on the HTTP/1.1 listener so the family overview dashboard can probe every node identically. Nothing in this repo probes a site's `/health/active` (Traefik health-probes central only), and there is **no `IHealthCheckPublisher` anywhere**, so those checks run only on an inbound HTTP probe. Consequence worth knowing: the standby's correct-by-design `active-node` Unhealthy verdict is logged by `DefaultHealthCheckService` at **Error** on every probe, which `LoggerConfigurationFactory` now suppresses with one narrowly-keyed Serilog filter (register row 71 — see `Component-Host.md` → *Standby health-check log noise*; genuine failures of every other check still log at Error). Central reaches a site by dialling `GrpcNodeAAddress`/`GrpcNodeBAddress` explicitly and flipping on error; whichever node answers, the site-internal `ClusterSingletonProxy` lands the work on the active node for free, so central still never needs to track which *site* node is active.
- **Seed-node ordering: every node lists ITSELF first (decision 2026-07-22) — the boot-alone gap is CLOSED.** Only `seed-nodes[0]` may self-join to form a new cluster (Akka runs `FirstSeedNodeProcess` for it, `JoinSeedNodeProcess` — which can never form one — for everyone else). All 14 shipped node appsettings now lead with the node's own address, so any node can cold-start alone and become operational unattended (~5s, `seed-node-timeout`); `StartupValidator` fails the boot if the ordering is broken (compares host AND port; Akka does no DNS canonicalisation). Two nodes cold-starting together while mutually reachable converge on ONE cluster via the `InitJoin` handshake — they split only under a genuine boot-time partition, the same class auto-down accepts. **An external self-form timer (`Cluster.Join(SelfAddress)` after a window) was implemented and REJECTED:** it sits outside the join handshake, so on a routine standby restart — where the peer is alive but the join is stalled behind removal of the node's own stale incarnation — it fires mid-join and permanently splits the pair (measured: still split after 90s). Regression tests: `SelfFirstSeedBootstrapTests`. The keep-oldest active-crash total outage was separately closed by the auto-down decision. See `docs/requirements/Component-ClusterInfrastructure.md` → Seed Node Ordering.
- **Simultaneous-cold-start split-brain guard (opt-in, Gitea #33) — the residual self-first cost, closed.** Self-first-on-both means a *truly simultaneous* cold start (shared power/hypervisor event) races `FirstSeedNodeProcess` on BOTH nodes → two 1-node clusters that never merge (in-process loopback tests converge and hide it; real parallel VM starts don't — OtOpcUa reproduced it live). Ported from OtOpcUa (`lmxopcua` `d1dac87f`): a **dark switch `ScadaBridge:Cluster:BootstrapGuard:Enabled` (default OFF, guard-off byte-identical)**. On: `BuildHocon` emits an EMPTY seed list (Akka does not auto-join) and `ClusterBootstrapCoordinator` (`IHostedService`, registered in BOTH the Central and Site composition roots) issues ONE reachability-gated `JoinSeedNodes` — the pure `ClusterBootstrapGuard` core makes the lower canonical `host:port` the **founder** (self-first, forms immediately), the higher node TCP-probes the founder up to `PartnerProbeSeconds` (25s) and joins **peer-first** if reachable else **self-first** (cold-start-alone preserved). Decided BEFORE a single join, **never re-forms mid-handshake** (that was the rejected self-form-timer's flaw); case-insensitive founder tie-break; probe timings validated `>0` when enabled. Residual accepted trade: founder dies in the probe→join window → higher node hangs unjoined, coordinator WARNs, a restart recovers it. Tests: `ClusterBootstrapGuardTests` (pure) + `ClusterBootstrapCoordinatorTests` (real-ActorSystem, incl. both-cold-start-together-form-one-cluster). **ENABLED on the docker rig (all 8 nodes) 2026-08-02** after `deploy.sh`'s simultaneous recreate split site pairs twice on 2026-08-01 with the guard off; live gate PASS — two consecutive simultaneous-start trials (a full redeploy + a full-topology `docker compose restart`), all four pairs converged (every lower-address node founded self-first, every higher node probed-then-joined peer-first), zero splits. Default remains OFF elsewhere (wonder-app-vd03, docker-env2, Host defaults). See `docs/requirements/Component-ClusterInfrastructure.md` → Simultaneous cold start.
@@ -190,7 +190,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
- Debug view: real-time streaming via DebugStreamBridgeActor + gRPC (events via `SiteStreamGrpcClient`, snapshot via the `SiteCommandService` query surface — `QueryReply.DebugViewSnapshot`). Health dashboard: 10s polling timer. Deployment status: real-time push via SignalR.
- Health reports: 30s interval, 60s offline threshold, monotonic sequence numbers, raw error counts per interval.
- Dead letter monitoring as a health metric.
- Site Event Logging: 30-day retention, 1GB storage cap, daily purge, paginated queries with keyword search.
- Site Event Logging: 30-day retention, 1GB storage cap, daily purge, paginated queries with keyword search. **The cap measures `site_events` + its four indexes, NOT the consolidated LocalDb file** (register row 81, 2026-08-17): whole-file measurement made the trim unwinnable on a replicated node — each batch freed event pages while adding an equal count of oplog rows + tombstones to the same file, so it stopped 2.5x over a 64 MB cap and a node with a down peer could not enforce the cap at all. `dbstat` (the exact per-table measurement) is **NOT compiled into the bundled `SQLitePCLRaw.lib.e_sqlite3`** — probed and pinned by `SqliteCapabilityTests`, so don't reach for it — the footprint is modelled from sampled row width instead (−0.8%…+12.8%, biased high). Consequence: `MaxStorageMb` no longer bounds the data volume alone; LocalDb bookkeeping is separate overhead bounded by `MaxOplogRows`/`TombstoneRetention` and is deliberately not this component's job. Decision: `docs/plans/2026-08-17-site-event-log-cap-semantics-decision.md`.
- Notification Outbox KPIs are central-computed point-in-time from the `Notifications` table (global + per-source-site): queue depth, stuck count, parked count, delivered-last-interval, oldest-pending age.
- Stuck = `Pending`/`Retrying` older than a configurable age threshold (default 10 min) — display-only (KPI count + row badge), no escalation/alerting.
- Headline KPI tiles surface on the Health dashboard; a new Central UI Notification Outbox page offers a queryable list with Retry/Discard actions on parked notifications.
code-reviews/NotificationOutbox/findings.md
+19 -1
View File
@@ -229,7 +229,25 @@ desired — capture the returned task and attach a continuation that calls
**Resolution**
_Unresolved._
Resolved — the preferred remedy was applied, but the resolution note was never
written back (the row was flipped to `Status: Resolved` while this body still read
`_Unresolved._`); recorded here during the deferred-register row 49 scoped-lifetime
sweep, which re-verified the fix against current code. Both helpers are now `async
Task` (`EmitAttemptAuditAsync` / `EmitTerminalAuditAsync`) and **await**
`_auditWriter.WriteAsync(evt)` inside the existing `try`/`catch`
(`NotificationOutboxActor.cs:731`, `:785`), so (1) the audit task can no longer
outlive the dispatcher's per-sweep DI scope, (2) the catch is reachable for faults
originating in the awaited body — a writer fault now reaches
`_logger.LogWarning` instead of being silently lost as an unobserved task, and
(3) the audit-write-never-affects-delivery invariant is preserved by the surrounding
catch rather than by the discard. The misleading "PipeTo is not used because the
writer never throws" XML-doc is gone; both helpers now carry a `<remarks>` block
stating the await-inside-try/catch rationale explicitly. Note this is a
*scope-too-narrow* hazard, the inverse of the *scope-too-wide* one in
NotificationOutbox-006 — the two are the opposite failure modes of the same
per-sweep-scope boundary, and both are now resolved. `CentralAuditWriter.WriteAsync`
still opens its own `CreateAsyncScope` (`CentralAuditWriter.cs:127-128`), so the
writer remains defensively scope-independent as a second line of defence.
### NotificationOutbox-005 — Ingest persistence inherits the CD-015 check-then-act race; under contention the second writer throws and the site retries
docker-env2/central-node-a/appsettings.Central.json
+3 -1
View File
@@ -4,7 +4,9 @@
"Role": "Central",
"NodeName": "central-a",
"NodeHostname": "scadabridge-env2-central-a",
"RemotingPort": 8081
"RemotingPort": 8081,
"_metricsPort": "Register row 56 — mirrors docker/central-node-a. Dedicated HTTP/1.1 Prometheus scrape listener; central binds it only when this key is present. In-container port only, no host mapping.",
"MetricsPort": 8084
},
"Cluster": {
"SeedNodes": [
docker-env2/central-node-b/appsettings.Central.json
+3 -1
View File
@@ -4,7 +4,9 @@
"Role": "Central",
"NodeName": "central-b",
"NodeHostname": "scadabridge-env2-central-b",
"RemotingPort": 8081
"RemotingPort": 8081,
"_metricsPort": "Register row 56 — mirrors docker/central-node-a. Dedicated HTTP/1.1 Prometheus scrape listener; central binds it only when this key is present. In-container port only, no host mapping.",
"MetricsPort": 8084
},
"Cluster": {
"SeedNodes": [
docker-env2/deploy.sh
+8
View File
@@ -3,6 +3,14 @@ set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
# Same guard as docker/deploy.sh: relative bind mounts mean a worktree deploy silently
# rehomes every node's data directory to the worktree's copy, splitting rig state.
if [[ "$SCRIPT_DIR" == *"/.claude/worktrees/"* && "${SCADABRIDGE_ALLOW_WORKTREE_DEPLOY:-}" != "1" ]]; then
echo "ERROR: refusing to deploy from a git worktree ($SCRIPT_DIR)." >&2
echo "Deploy from the main checkout, or set SCADABRIDGE_ALLOW_WORKTREE_DEPLOY=1 to override." >&2
exit 1
fi
echo "=== ScadaBridge Env2 Docker Deploy ==="
# Reuse the primary build (same scadabridge:latest image, same network creation)
docker-env2/docker-compose.yml
+8
View File
@@ -70,6 +70,10 @@ services:
container_name: scadabridge-env2-site-x-a
environment:
SCADABRIDGE_CONFIG: Site
# Register row 63 — mirrors docker/docker-compose.yml. Enables the Development-only
# DI scope validation (register row 50) on the site nodes, which previously ran
# unvalidated because only the central containers set this. Dev/rig only.
ASPNETCORE_ENVIRONMENT: Development
ports:
- "9121:8082" # Akka remoting
- "9123:8083" # gRPC streaming
@@ -90,6 +94,10 @@ services:
container_name: scadabridge-env2-site-x-b
environment:
SCADABRIDGE_CONFIG: Site
# Register row 63 — mirrors docker/docker-compose.yml. Enables the Development-only
# DI scope validation (register row 50) on the site nodes, which previously ran
# unvalidated because only the central containers set this. Dev/rig only.
ASPNETCORE_ENVIRONMENT: Development
ports:
- "9122:8082" # Akka remoting
- "9124:8083" # gRPC streaming
docker/README.md
+63
View File
@@ -87,6 +87,7 @@ docker/
├── build.sh # Build Docker image
├── deploy.sh # Build + deploy all containers
├── seed-sites.sh # Create test sites with Akka + gRPC addresses
├── seed-live-probe.sh # Seed one deployable live-data instance on site-a (stream traffic)
├── teardown.sh # Stop and remove containers
├── central-node-a/
│ ├── appsettings.Central.json # Central node A configuration
@@ -160,8 +161,70 @@ docker/deploy.sh
# 3. Seed test sites (first-time only, after cluster is healthy)
docker/seed-sites.sh
# 4. Seed one deployable live-data instance on site-a, so the rig has real
# stream traffic for live probes (see "Live-data probe instance" below)
docker/seed-live-probe.sh
```
### Live-data probe instance
`seed-live-probe.sh` creates the minimum working set the rig needs for any live probe
of the data or streaming path: a `RigProbe` template with two attributes bound to
**self-changing** opc-plc nodes (`ns=3;s=SpikeData`, `ns=3;s=RandomSignedInt32`, both
~1 Hz) plus a 5 s heartbeat script, and a deployed `rig-probe-1` instance on site-a.
Two traps it exists to avoid:
- **The custom tags in `infra/opcua/nodes.json` are declared but not simulated.**
`Motor.*`, `Tank.*`, `Valve.*`, `TestChildObject.*` read 0 / false / epoch forever.
Binding to them gives a `Connected` data connection with `Good` quality and *zero*
event traffic, which looks healthy and probes nothing. Use the built-in simulated
nodes (`SpikeData`, `RandomSignedInt32`, `StepUp`, `FastUInt1..5`, `SlowUInt1..5`).
- **`scadabridge_site_stream_subscribers_active` is not data-connection health.** It counts
open central→site gRPC *subscriber streams*, so it sits at 0 on a perfectly healthy site
with nobody watching. (Register row 54 renamed it from `scadabridge_site_connection_up`,
whose name invited exactly that misreading — it is what made row 48's "0 on all six site
nodes" look like a fault when it was healthy steady state.)
For real data-connection health, the same row added two honest gauges:
| Series | Meaning |
| --- | --- |
| `scadabridge_site_data_connections_connected` | Connections currently in the `Connected` state |
| `scadabridge_site_data_connections_total` | Connections known to this node, in any state |
`connected < total` is the alert condition — neither number alone separates "a device is
down" from "nothing is configured". Both read **0 on the standby node**, which is healthy:
connections are opened by the deployment path on the active node only, so alert on the
active node or on `max()` across the pair. The health report's `dataConnectionEndpoints` /
`dataConnectionTagQuality` (via `scadabridge health site`) remain the per-connection detail.
Verify (metrics are on container port 8084 — **all eight nodes** as of register row 56, which
gave the two central nodes the same dedicated scrape listener the six site nodes already had;
the containers ship no curl/wget):
```bash
scadabridge debug snapshot --id <instance-id>
scadabridge health site --identifier site-a
scadabridge debug stream --id <instance-id> # open a stream, then in another shell:
docker exec scadabridge-site-a-a bash -c \
'exec 3<>/dev/tcp/localhost/8084 && printf "GET /metrics HTTP/1.0\r\nHost: localhost\r\n\r\n" >&3 && cat <&3' \
| grep -E 'stream_subscribers_active|stream_batch_size|data_connections'
# Central-only series (KPI/outbox counters, bundle-import session gauge + rejection
# counter, alarm-cache gauges) — same recipe, central container:
docker exec scadabridge-central-a bash -c \
'exec 3<>/dev/tcp/localhost/8084 && printf "GET /metrics HTTP/1.0\r\nHost: localhost\r\n\r\n" >&3 && cat <&3' \
| grep -E 'import_sessions|alarm_cache'
```
> Central binds 8084 only because `docker/central-node-*/appsettings.Central.json` now sets
> `ScadaBridge:Node:MetricsPort`. Unlike a site node, central reads that key **only when
> present** — a central deployment without it is unchanged and still serves `/metrics` on its
> web port, since the endpoint rides endpoint routing and is served on every bound listener.
### After Code Changes
Rebuild and redeploy. The Docker build cache skips NuGet restore when only source files change:
docker/central-node-a/appsettings.Central.json
+3 -1
View File
@@ -4,7 +4,9 @@
"Role": "Central",
"NodeName": "central-a",
"NodeHostname": "scadabridge-central-a",
"RemotingPort": 8081
"RemotingPort": 8081,
"_metricsPort": "Register row 56. Central maps /metrics on every listener it binds, so it was always reachable on the web port (:5000) — but not on 8084, which is the port the live-scrape recipe probes (docker exec ... </dev/tcp/localhost/8084). Declaring it here binds the same dedicated HTTP/1.1 scrape listener the six site nodes use, so one probe recipe works on all eight nodes and central-only series (KPI/outbox counters, the import-session gauge + rejection counter, alarm-cache gauges) are scrapeable live. Central reads this key ONLY when present — unlike a site node, which always binds 8084 — so an existing central without it is unchanged. No host port mapping is needed: the probe runs inside the container.",
"MetricsPort": 8084
},
"Cluster": {
"SeedNodes": [
docker/central-node-b/appsettings.Central.json
+3 -1
View File
@@ -4,7 +4,9 @@
"Role": "Central",
"NodeName": "central-b",
"NodeHostname": "scadabridge-central-b",
"RemotingPort": 8081
"RemotingPort": 8081,
"_metricsPort": "Register row 56 — see central-node-a. Dedicated HTTP/1.1 Prometheus scrape listener, matching the six site nodes so one probe recipe covers all eight. Central binds this only when the key is present.",
"MetricsPort": 8084
},
"Cluster": {
"SeedNodes": [
docker/deploy.sh
+11
View File
@@ -3,6 +3,17 @@ set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
# The compose file's bind mounts are RELATIVE (./site-a-node-a/data etc.), so running
# this script from a git worktree silently rehomes every node's data directory to the
# worktree's copy — the rig keeps running, but its SQLite stores split from the main
# checkout's and all state written there is lost on the next deploy from main.
if [[ "$SCRIPT_DIR" == *"/.claude/worktrees/"* && "${SCADABRIDGE_ALLOW_WORKTREE_DEPLOY:-}" != "1" ]]; then
echo "ERROR: refusing to deploy from a git worktree ($SCRIPT_DIR)." >&2
echo "Bind-mount data dirs would rehome to the worktree copy and split rig state." >&2
echo "Deploy from the main checkout, or set SCADABRIDGE_ALLOW_WORKTREE_DEPLOY=1 to override." >&2
exit 1
fi
echo "=== ScadaBridge Docker Deploy ==="
# Build image (creates network if needed)
docker/docker-compose.yml
+48
View File
@@ -213,6 +213,14 @@ services:
# file). site-b and site-c deliberately carry neither.
<<: [*secrets-store-env, *secrets-hub-env, *secrets-hub-site-env]
SCADABRIDGE_CONFIG: Site
# Register row 63. The central containers have always set this; the site containers
# never did, so the Development-only DI scope validation (register row 50,
# ServiceProviderValidation.Configure — ValidateScopes/ValidateOnBuild) was dead on
# exactly the half of the topology that has six nodes. The site graph now validates
# clean, so turning it on here buys live coverage: a captive dependency or an
# unconstructible site-only descriptor fails the container at boot instead of
# surfacing as a mid-run resolve error. Dev/rig only — never set in production.
ASPNETCORE_ENVIRONMENT: Development
ports:
- "9021:8082" # Akka remoting (host access for debugging)
- "9023:8083" # gRPC streaming
@@ -241,6 +249,14 @@ services:
# file). site-b and site-c deliberately carry neither.
<<: [*secrets-store-env, *secrets-hub-env, *secrets-hub-site-env]
SCADABRIDGE_CONFIG: Site
# Register row 63. The central containers have always set this; the site containers
# never did, so the Development-only DI scope validation (register row 50,
# ServiceProviderValidation.Configure — ValidateScopes/ValidateOnBuild) was dead on
# exactly the half of the topology that has six nodes. The site graph now validates
# clean, so turning it on here buys live coverage: a captive dependency or an
# unconstructible site-only descriptor fails the container at boot instead of
# surfacing as a mid-run resolve error. Dev/rig only — never set in production.
ASPNETCORE_ENVIRONMENT: Development
ports:
- "9022:8082" # Akka remoting
- "9024:8083" # gRPC streaming
@@ -267,6 +283,14 @@ services:
environment:
<<: *secrets-store-env
SCADABRIDGE_CONFIG: Site
# Register row 63. The central containers have always set this; the site containers
# never did, so the Development-only DI scope validation (register row 50,
# ServiceProviderValidation.Configure — ValidateScopes/ValidateOnBuild) was dead on
# exactly the half of the topology that has six nodes. The site graph now validates
# clean, so turning it on here buys live coverage: a captive dependency or an
# unconstructible site-only descriptor fails the container at boot instead of
# surfacing as a mid-run resolve error. Dev/rig only — never set in production.
ASPNETCORE_ENVIRONMENT: Development
ports:
- "9031:8082" # Akka remoting
- "9033:8083" # gRPC streaming
@@ -293,6 +317,14 @@ services:
environment:
<<: *secrets-store-env
SCADABRIDGE_CONFIG: Site
# Register row 63. The central containers have always set this; the site containers
# never did, so the Development-only DI scope validation (register row 50,
# ServiceProviderValidation.Configure — ValidateScopes/ValidateOnBuild) was dead on
# exactly the half of the topology that has six nodes. The site graph now validates
# clean, so turning it on here buys live coverage: a captive dependency or an
# unconstructible site-only descriptor fails the container at boot instead of
# surfacing as a mid-run resolve error. Dev/rig only — never set in production.
ASPNETCORE_ENVIRONMENT: Development
ports:
- "9032:8082" # Akka remoting
- "9034:8083" # gRPC streaming
@@ -319,6 +351,14 @@ services:
environment:
<<: *secrets-store-env
SCADABRIDGE_CONFIG: Site
# Register row 63. The central containers have always set this; the site containers
# never did, so the Development-only DI scope validation (register row 50,
# ServiceProviderValidation.Configure — ValidateScopes/ValidateOnBuild) was dead on
# exactly the half of the topology that has six nodes. The site graph now validates
# clean, so turning it on here buys live coverage: a captive dependency or an
# unconstructible site-only descriptor fails the container at boot instead of
# surfacing as a mid-run resolve error. Dev/rig only — never set in production.
ASPNETCORE_ENVIRONMENT: Development
ports:
- "9041:8082" # Akka remoting
- "9043:8083" # gRPC streaming
@@ -345,6 +385,14 @@ services:
environment:
<<: *secrets-store-env
SCADABRIDGE_CONFIG: Site
# Register row 63. The central containers have always set this; the site containers
# never did, so the Development-only DI scope validation (register row 50,
# ServiceProviderValidation.Configure — ValidateScopes/ValidateOnBuild) was dead on
# exactly the half of the topology that has six nodes. The site graph now validates
# clean, so turning it on here buys live coverage: a captive dependency or an
# unconstructible site-only descriptor fails the container at boot instead of
# surfacing as a mid-run resolve error. Dev/rig only — never set in production.
ASPNETCORE_ENVIRONMENT: Development
ports:
- "9042:8082" # Akka remoting
- "9044:8083" # gRPC streaming
docker/seed-live-probe.sh
Executable
+125
View File
@@ -0,0 +1,125 @@
#!/bin/bash
set -euo pipefail
# Seed ONE deployable, live-data instance on site-a so the rig always has real
# stream traffic for live probes.
#
# Why this exists: the rig repeatedly ends up with instances that cannot deploy
# (deferred-work register row 48 — the seven `soak-*` rows fail pre-deployment
# validation because their bound data connections no longer exist on the sites).
# With nothing deployed, `debug snapshot` returns `instanceNotFound`, no gRPC site
# stream is ever opened, and the R2 batching histogram
# `scadabridge.site.stream.batch_size` has nothing to record — so any live probe of
# the streaming path is unverifiable. This script rebuilds the minimum working set
# from scratch, using only the CLI (no DB edits).
#
# What it creates (all on site-a / site id 1, against data connection id 1
# "OPC PLC Simulator" -> opc.tcp://scadabridge-opcua:50000):
# - Template `RigProbe`
# * Spike (Double) <- ns=3;s=SpikeData — opc-plc built-in, changes ~1/s
# * RandomInt (Int32) <- ns=3;s=RandomSignedInt32 — opc-plc built-in, changes ~1/s
# * Heartbeat (Int32) — static, bumped every 5s by the Heartbeat script
# * Heartbeat script — Interval 5000ms, trust-policy-clean
# - Instance `rig-probe-1`, bound + deployed
#
# IMPORTANT — pick self-changing tags. The custom tags in `infra/opcua/nodes.json`
# (Motor.*, Tank.*, Valve.*, TestChildObject.*) are declared but NOT simulated by
# opc-plc: they read 0 / false / epoch and never change on their own, so binding to
# them yields a healthy connection with zero event traffic. The built-in simulated
# nodes are the ones that move: SpikeData, RandomSignedInt32, StepUp, FastUInt1..5
# (1s) and SlowUInt1..5 (10s).
#
# Prerequisites:
# - Infrastructure services running (infra/docker-compose.yml — needs scadabridge-opcua)
# - Application containers running (docker/deploy.sh)
# - Sites seeded (docker/seed-sites.sh) and central healthy
# curl -sf http://localhost:9000/health/ready
#
# Usage:
# docker/seed-live-probe.sh
#
# Idempotency: this script is NOT idempotent — template and instance names must be
# unique, so a second run fails on the create steps. To re-seed, delete the previous
# instance and template first:
# scadabridge instance delete --id <id> && scadabridge template delete --id <id>
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
CLI="dotnet run --project $PROJECT_ROOT/src/ZB.MOM.WW.ScadaBridge.CLI --"
AUTH="--username multi-role --password password"
URL="--url http://localhost:9000"
SITE_ID=1 # site-a
CONNECTION_ID=1 # "OPC PLC Simulator" on site-a
TEMPLATE_NAME="RigProbe"
INSTANCE_NAME="rig-probe-1"
# Pulls the numeric "id" out of the CLI's JSON reply without needing jq.
json_id() { python3 -c 'import json,sys; print(json.load(sys.stdin)["id"])'; }
echo "=== Seeding live-data probe instance on site-a ==="
echo ""
echo "1/6 Creating template $TEMPLATE_NAME..."
TEMPLATE_ID=$($CLI $URL $AUTH template create \
--name "$TEMPLATE_NAME" \
--description "Minimal live-traffic probe: two OPC UA sim tags that change on their own + a 5s heartbeat script" \
| json_id)
echo " template id = $TEMPLATE_ID"
echo ""
echo "2/6 Adding data-bound attributes..."
$CLI $URL $AUTH template attribute add \
--template-id "$TEMPLATE_ID" --name Spike --data-type Double \
--data-source "ns=3;s=SpikeData" \
--description "opc-plc SpikeData — Double, changes every ~1s on its own" > /dev/null
$CLI $URL $AUTH template attribute add \
--template-id "$TEMPLATE_ID" --name RandomInt --data-type Int32 \
--data-source "ns=3;s=RandomSignedInt32" \
--description "opc-plc RandomSignedInt32 — Int32, changes every ~1s on its own" > /dev/null
echo ""
echo "3/6 Adding the static heartbeat attribute + its interval script..."
$CLI $URL $AUTH template attribute add \
--template-id "$TEMPLATE_ID" --name Heartbeat --data-type Int32 --value 0 \
--description "Static attribute bumped every 5s by the Heartbeat script — script-sourced traffic" > /dev/null
$CLI $URL $AUTH template script add \
--template-id "$TEMPLATE_ID" --name Heartbeat \
--trigger-type Interval --trigger-config '{"intervalMs":5000}' \
--code 'var current = await Instance.GetAttribute("Heartbeat"); var next = (int.TryParse(current?.ToString(), out var n) ? n : 0) + 1; await Instance.SetAttribute("Heartbeat", next.ToString());' > /dev/null
echo ""
echo "4/6 Creating instance $INSTANCE_NAME..."
INSTANCE_ID=$($CLI $URL $AUTH instance create \
--name "$INSTANCE_NAME" --template-id "$TEMPLATE_ID" --site-id "$SITE_ID" | json_id)
echo " instance id = $INSTANCE_ID"
echo ""
echo "5/6 Binding the data-sourced attributes to connection $CONNECTION_ID..."
$CLI $URL $AUTH instance set-bindings --id "$INSTANCE_ID" \
--bindings "[[\"Spike\",$CONNECTION_ID],[\"RandomInt\",$CONNECTION_ID]]" > /dev/null
echo ""
echo "6/6 Deploying..."
$CLI $URL $AUTH instance deploy --id "$INSTANCE_ID"
echo ""
echo "=== Done. Verify with: ==="
echo " scadabridge debug snapshot --id $INSTANCE_ID"
echo " scadabridge health site --identifier site-a # dataConnectionEndpoints / dataConnectionTagQuality"
echo " scadabridge debug stream --id $INSTANCE_ID # opens a gRPC site stream; ~2 events/s"
echo ""
echo "While a stream is open, the site node serving it reports both signals"
echo "(metrics are on container port 8084, and the containers have no curl/wget):"
echo " docker exec scadabridge-site-a-a bash -c 'exec 3<>/dev/tcp/localhost/8084 && printf \"GET /metrics HTTP/1.0\\r\\nHost: localhost\\r\\n\\r\\n\" >&3 && cat <&3' \\"
echo " | grep -E 'stream_subscribers_active|stream_batch_size|data_connections'"
echo ""
echo "NOTE: scadabridge_site_stream_subscribers_active (renamed from the misleading"
echo "scadabridge_site_connection_up, register row 54) counts OPEN CENTRAL->SITE gRPC"
echo "SUBSCRIBER STREAMS, not OPC UA data connections. It reads 0 on a perfectly healthy"
echo "site with no subscriber attached — that is correct, not a fault."
echo ""
echo "For real data-connection health use the gauges added by the same row:"
echo " scadabridge_site_data_connections_connected / _total"
echo "connected < total is the alert condition. Both read 0 on the STANDBY node, since"
echo "connections are opened by the deployment path on the active node only."
docs/components/Commons.md
+1 -1
View File
@@ -295,7 +295,7 @@ public record AlarmStateChanged(
### Observability
`Observability/ScadaBridgeTelemetry` defines the singleton `Meter` named `ZB.MOM.WW.ScadaBridge` and the application-wide instrument definitions. Components call the static emit helpers (`RecordDeploymentApplied`, `SiteConnectionOpened`, etc.) rather than creating their own meters. Instruments are no-ops until an OTel listener attaches, so uninstrumented hosts pay no overhead.
`Observability/ScadaBridgeTelemetry` defines the singleton `Meter` named `ZB.MOM.WW.ScadaBridge` and the application-wide instrument definitions. Components call the static emit helpers (`RecordDeploymentApplied`, `SiteStreamSubscriberOpened`, etc.) rather than creating their own meters. Instruments are no-ops until an OTel listener attaches, so uninstrumented hosts pay no overhead.
## Usage
docs/components/ConfigurationDatabase.md
+5 -3
View File
@@ -7,7 +7,7 @@ The Configuration Database component is the exclusive EF Core data-access layer
The component lives in `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/` and is central-only — site clusters never load it. Its responsibilities break down into four areas:
- **DbContext + Fluent mappings** — `ScadaBridgeDbContext` maps ~30 Commons POCO entity types to SQL Server using `IEntityTypeConfiguration<T>` classes in `Configurations/`, registered wholesale via `modelBuilder.ApplyConfigurationsFromAssembly(...)`.
- **Repository implementations** — eleven scoped repositories implement the interfaces declared in Commons, covering every domain area from template authoring to audit log ingest.
- **Repository implementations** — scoped repositories implement the interfaces declared in Commons, covering every domain area from template authoring to audit log ingest.
- **Config-change audit** — `AuditService` implements `IAuditService`, staging an `AuditLogEntry` into the change tracker so it commits atomically with the entity change; `AuditCorrelationContext` threads a `BundleImportId` through `AsyncLocal<T>` so bundle-import audit rows are correlated without cross-contaminating concurrent import sessions.
- **Partition maintenance** — `AuditLogPartitionMaintenance` implements `IPartitionMaintenance`, rolling `pf_AuditLog_Month` forward by issuing `ALTER PARTITION FUNCTION … SPLIT RANGE` for each missing future monthly boundary.
@@ -91,9 +91,11 @@ The predecessor `UX_AuditLog_EventId` — a single-column unique index on `[PRIM
**`SiteCallEntityTypeConfiguration`** maps `SiteCall` to `dbo.SiteCalls` with a `TrackedOperationId` PK stored as `varchar(36)` (GUID in `"D"` format) so the column shape matches the wire format and the site SQLite store — one consistent format for operational debugging.
**`BundleUnlockAttemptEntityTypeConfiguration`** maps `BundleUnlockAttempt` to the table of the same name — the fleet-wide sliding window behind Transport's per-client bundle-unlock throttle (register row 52; the counters used to be per-node and per-process, so the effective cap was N × the configured limit). Operational history, not audit: non-partitioned, `[PRIMARY]` filegroup, no DB-role restriction. One named index, `IX_BundleUnlockAttempt_Client_Attempted`, serves both the windowed count and the application's opportunistic prune, which is bounded and best-effort — a store failure in this path is deliberately fail-open at the Transport layer, so nothing here may block an unlock. The table deliberately records no bundle identity; the durable record of a failed unlock is the `BundleImportUnlockFailed` audit row.
### Repository implementations
All eleven repositories follow the same shape: they take `ScadaBridgeDbContext` by constructor injection, work with Commons POCO types, and never commit — callers invoke `SaveChangesAsync()` to commit the unit of work.
All repositories follow the same shape: they take `ScadaBridgeDbContext` by constructor injection, work with Commons POCO types, and never commit — callers invoke `SaveChangesAsync()` to commit the unit of work.
**`AuditLogRepository`** is the most specialized. Its `InsertIfNotExistsAsync` bypasses the change tracker and issues raw interpolated SQL because the computed columns must not appear in the INSERT column list:
@@ -152,7 +154,7 @@ services.AddConfigurationDatabase(
configuration["ScadaBridge:Database:ConfigurationDb"]!);
```
This registers `ScadaBridgeDbContext` as scoped (with the runtime encryption overload), all eleven repository interfaces bound to their implementations, `IAuditCorrelationContext` → `AuditCorrelationContext`, `IAuditService` → `AuditService`, `IInstanceLocator` → `InstanceLocator`, `IPartitionMaintenance` → `AuditLogPartitionMaintenance`, and the Data Protection key ring persisted to the database.
This registers `ScadaBridgeDbContext` as scoped (with the runtime encryption overload), every repository interface bound to its implementation, `IAuditCorrelationContext` → `AuditCorrelationContext`, `IAuditService` → `AuditService`, `IInstanceLocator` → `InstanceLocator`, `IPartitionMaintenance` → `AuditLogPartitionMaintenance`, and the Data Protection key ring persisted to the database.
The obsolete zero-argument overload throws `InvalidOperationException` at startup (marked `error: true` on the `[Obsolete]` attribute) so a misconfigured host fails fast with a clear message rather than silently producing an empty DI registration.
docs/components/DataConnectionLayer.md
+1 -1
View File
@@ -161,7 +161,7 @@ A separate in-flight set `_subscribesInFlight` tracks the initial `SubscribeAsyn
### Creating a connection
The Site Runtime's `DeploymentManagerActor` sends `CreateConnectionCommand` to `DataConnectionManagerActor` during instance deployment:
The Site Runtime's `DeploymentManagerActor` sends `CreateConnectionCommand` to `DataConnectionManagerActor` whenever it creates an `InstanceActor` — the config comes from the `Connections` map inside that instance's flattened config, not from the site's `data_connection_definitions` table (which has no production reader). The manager never reads storage and has no independent view of which connections a site should have, so a start path that fails to push is indistinguishable here from a connection that genuinely does not exist: that was register row **85**, where two of the four Instance-Actor start paths omitted the push and every subscribe from the resulting instances was answered `Unknown connection: <name>` forever. The push now lives inside `CreateInstanceActor` on the Site Runtime side. Sends are hash-guarded there, so an unchanged connection produces no message at all:
```csharp
// Commons/Messages/DataConnection/CreateConnectionCommand.cs
docs/components/SiteRuntime.md
+20 -5
View File
@@ -27,6 +27,17 @@ The component code lives in `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/`:
The singleton reads all deployed configurations from SQLite in `PreStart`, compiles shared scripts off-thread, and then creates `InstanceActor` children in batches. The default batch size is 20, with a 100 ms delay between batches (`StartupBatchSize` and `StartupBatchDelayMs` in `SiteRuntimeOptions`). Without staggering, 500 instances each subscribing to OPC UA tags simultaneously would produce a reconnection storm that overwhelms the OPC UA server.
### Instance reconcile sweep
The staggered startup above reads storage exactly once. Anything that writes a `deployed_configurations` row afterwards without going through a deploy command therefore changed the store but not the runtime — and nothing said so. Two real cases (deferred register rows **78** and **72**):
- a config seeded by **LocalDb CDC replication** (a fresh standby filled in from its peer, or a snapshot resync) landed after the boot load and was invisible to the startup reconcile, so its enabled instances never started;
- a config the startup reconcile **refreshed** because it was stale left the running `InstanceActor` serving the previously compiled revision.
`DeploymentManagerActor` therefore runs a periodic sweep (`InstanceSweepIntervalSeconds`, default 60; `0` disables) that re-reads the durable rows off-thread and, on the actor thread, for every **enabled** stored config either starts a missing `InstanceActor` or — when the running one's revision no longer matches the stored one — re-applies the stored config **through the ordinary redeploy path**, so nothing about redeploy semantics is re-implemented. It defers to any in-flight deploy/warm/termination, never starts a disabled instance, stays inert until boot-time creation has finished, makes one repair attempt per (instance, revision) pair so it can never become a restart loop, and is completely silent when there is nothing to repair. Repairs emit Warning `deployment` site events naming the trigger (and, for a stale-revision restart, both revisions). Design note: `docs/plans/2026-08-15-instance-sweep-self-heal-design.md`.
Register row **85** was the same gap one layer down: the sweep started the instance, but the instance came up with **no DCL connection actors**, so every tag subscribe was answered `Unknown connection: <name>`. The connection push (`EnsureDclConnections`) was the *caller's* job and two of the four Instance-Actor start paths omitted it; it now lives inside `CreateInstanceActor`. See *Deployment flow* below and §7 of the design note.
### Actor supervision and lifetimes
| Actor | Kind | Supervises children with | On exception |
@@ -74,10 +85,11 @@ Prior to WP3.1 (2026-08-15) each invocation spawned a short-lived `ScriptExecuti
Central sends a `DeployInstanceCommand` carrying a JSON `FlattenedConfiguration` to the site singleton. The singleton:
1. Calls `EnsureDclConnections` to push any new or changed connection definitions to the DCL manager (hash-guarded: unchanged configs are skipped).
2. Calls `CreateInstanceActor`, which does `Context.ActorOf(props, instanceName)`.
3. Runs an off-thread `Task` that calls `SiteStorageService.StoreDeployedConfigAsync` and clears static overrides and native alarm state. Nothing is pushed to the peer: as of LocalDb Phase 2 those three tables are replicated, so the writes themselves reach the standby.
4. Pipes back a `DeployPersistenceResult`; only on success does it tell the deployer `DeploymentStatus.Success`. If persistence fails, the optimistically-created actor is stopped and the error is returned to central (`SiteRuntime-005`).
1. Calls `CreateInstanceActor`, which first calls `EnsureDclConnections` to push any new or changed connection definitions to the DCL manager (hash-guarded: unchanged configs are skipped), then does `Context.ActorOf(props, instanceName)`.
2. Runs an off-thread `Task` that calls `SiteStorageService.StoreDeployedConfigAsync` and clears static overrides and native alarm state. Nothing is pushed to the peer: as of LocalDb Phase 2 those three tables are replicated, so the writes themselves reach the standby.
3. Pipes back a `DeployPersistenceResult`; only on success does it tell the deployer `DeploymentStatus.Success`. If persistence fails, the optimistically-created actor is stopped and the error is returned to central (`SiteRuntime-005`).
> **`CreateInstanceActor` is the single funnel for the connection push too (register row 85)**: the `EnsureDclConnections` call moved *into* `CreateInstanceActor` on 2026-08-17. It used to sit at each caller, and only two of the four callers made it — the boot-time batch and this deploy path. `StartStoredInstance` (row-67 ensure-started and the row-78 sweep) and `HandleEnableResult` did not, so a self-healed or newly-enabled instance came up with no connection actors and every tag subscribe was answered `Unknown connection: <name>` indefinitely. See the *Instance reconcile sweep* section and `docs/plans/2026-08-15-instance-sweep-self-heal-design.md` §7.
For redeployment (instance already running), the existing actor is stopped and watched:
@@ -205,7 +217,7 @@ Both `AlarmActor` and `NativeAlarmActor` tell the `InstanceActor` an `AlarmState
| `static_attribute_overrides` | Runtime attribute writes (`SetAttribute` on static attrs) | Yes — cleared by `ClearStaticOverridesAsync` |
| `native_alarm_state` | Mirrored native alarm conditions (survives failover) | Yes — cleared by `ClearNativeAlarmsForInstanceAsync` |
| `shared_scripts` | Shared script code from artifact deployments | No |
| `external_systems` | External system definitions | No |
| `external_systems` | External system definitions (incl. `timeout_seconds`, and `max_retries`/`retry_delay_ms` — the deployed store-and-forward retry policy, `0` = use the engine default) | No |
| `database_connections` | Database connection strings | No |
| `data_connection_definitions` | OPC UA / MxGateway endpoint definitions | No |
| `notification_lists` | Notification list definitions | No |
@@ -215,6 +227,8 @@ Both `AlarmActor` and `NativeAlarmActor` tell the `InstanceActor` an `AlarmState
Config replication has no actor. `SiteReplicationActor` — which received `ReplicateConfigDeploy` / `ReplicateConfigRemove` / `ReplicateConfigSetEnabled` / `ReplicateArtifacts` / `ReplicateStoreAndForward` from the active node's `DeploymentManagerActor`, tracked the peer through cluster membership events, and forwarded each command to `/user/site-replication` via `ActorSelection` — was deleted in LocalDb Phase 2, together with its notify-and-fetch exchange (the standby was told a deploy had happened and then HTTP-fetched the config itself). The site's config tables are now replicated by LocalDb CDC, so a deploy on either node reaches the other as an ordinary row change. `SiteReconciliationActor` still fetches over HTTP at node startup when central reports gaps; that is a different path and it survives.
Since register row 67 that pass also heals the **runtime**, not only the file. It runs after `DeploymentManagerActor`'s boot-time SQLite load, so a config it stored used to start no instance actor — an enabled instance missing at boot needed a *second* restart to run. Each enabled instance it heals now gets a fire-and-forget `EnsureInstanceStartedCommand` sent through the deployment-manager singleton proxy; the handler creates the instance actor only if one is not already running, re-reads the stored row to honour `is_enabled`, and steps aside for any in-flight deploy. A start emits a Warning log line and a `deployment` site event so the heal is visible.
## Usage
### Lifecycle commands
@@ -263,6 +277,7 @@ All options live in the `ScadaBridge:SiteRuntime` section, bound to `SiteRuntime
| `MaxConcurrentRunsPerScript` | `4` | Runs in flight (queued or executing) per script/alarm on-trigger script before the newest is shed (WP3.1) |
| `MirroredAlarmCapPerSource` | `1000` | Maximum mirrored conditions per `NativeAlarmActor` source binding before oldest is dropped and logged |
| `NativeAlarmRetryIntervalMs` | `5000` | Milliseconds before retrying a failed native alarm subscription |
| `InstanceSweepIntervalSeconds` | `60` | Interval of the singleton's instance reconcile sweep (register rows 78 + 72); `0` disables it — see Instance reconcile sweep |
The SQLite connection string is passed directly to `AddSiteRuntime(connectionString)` in the host composition root and is not part of `SiteRuntimeOptions`.
docs/components/StoreAndForward.md
+2
View File
@@ -51,6 +51,8 @@ The retry interval is fixed — not exponential. There is no maximum buffer size
`StoreAndForwardMessage.MaxRetries` controls how many background-sweep attempts the engine makes before parking. `MaxRetries = 0` means **no limit** — the message retries on every sweep until delivered and is never parked for retry exhaustion. It is not a "never retry" value; callers that want unbounded retry pass `maxRetries: 0` explicitly. The `EnqueueAsync` `maxRetries` parameter defaults to `StoreAndForwardOptions.DefaultMaxRetries` (50).
**The source-entity `0` means the opposite, and the gateways convert.** On an `ExternalSystemDefinition` / `DatabaseConnectionDefinition`, `MaxRetries = 0` (and `RetryDelay = TimeSpan.Zero`) mean *unset*, not *unbounded* — so `ExternalSystemClient.CachedCallAsync` and `DatabaseGateway` pass `system.MaxRetries > 0 ? system.MaxRetries : null`, letting the bounded engine default apply. Forwarding the entity's `0` verbatim would make every unconfigured cached call retry forever. The external system's values only actually reach a site as of deferred-work row 27; before that the artifact did not carry them.
### Messages not cleared on instance deletion
When an instance is deleted, its buffered S&F messages are not removed. `StoreAndForwardMessage.OriginInstanceName` records the originating instance at enqueue time so the buffer can continue to deliver and so the central UI can attribute parked messages even after the instance is gone.
docs/components/Transport.md
+19 -3
View File
@@ -49,6 +49,14 @@ After `LoadAsync` validates and decrypts a bundle, the plaintext content is stor
Without this, an abandoned wizard or a plain CLI preview held one of the eight `MaxConcurrentImportSessions` slots — and the decrypted bundle content — for the full TTL; eight of them locked out every import on the active node (deferred register row 43, observed live).
**Session-cap observability (register row 53).** That leak was only ever inferable from rotating test failures, because nothing counted open sessions and nothing logged a rejection. Two signals now close the gap:
- **Gauge `scadabridge.central.transport.import_sessions.open`** — open (unexpired) sessions on this node, on the `ZB.MOM.WW.ScadaBridge` meter, which is already in `SiteServiceRegistration.ObservedMeters` (the `ZbTelemetryOptions.Meters` allowlist; an unlisted meter exports nothing, silently). The provider is registered by `BundleSessionEvictionService.StartAsync` and identity-checked-cleared on `StopAsync`, mirroring the StoreAndForward queue-depth gauge. `IBundleSessionStore.OpenSessionCount` deliberately excludes expired-but-unswept entries — counting the raw dictionary would report the node as pinned at the cap for up to a sweep interval after a burst had actually drained, which is the exact false positive this instrument exists to rule out. Transport is registered only in the host's Central branch, so the gauge reports 0 on a site node.
- **Counter `scadabridge.central.transport.import_sessions.rejected`** (register row 66) — bundle-import sessions turned away because the cap was exhausted, on the same meter (so likewise no allowlist change). Row 53 shipped the gauge and a Warning log, but a log line cannot be alerted on; this is the series to alert on. It is incremented **only at the post-eviction recheck** in `BundleSessionStore.Open`: the cap is tested twice, and a store full of merely *expired* sessions reclaims and admits the caller, so counting the first test would fire the signal routinely on any node whose sweep has not yet run. Re-`Open`ing an already-open session id skips the cap block entirely and is likewise never counted.
Read the two instruments **together** — the pair is what discriminates the failure modes. Rejections climbing while the gauge oscillates *below* the cap is genuine concurrency: raise `MaxConcurrentImportSessions`. Rejections climbing while the gauge sits *pinned at* the cap is a session leak (the register row 43 signature); raising the cap only defers it, so find the leak instead.
- **Warning log on rejection** — `BundleSessionStore.Open` logs the open count, the cap, and the TTL when it rejects. It fires only on a real cap hit: a store full of *expired* sessions is reclaimed first and logs nothing. A count that stays pinned at the cap across repeated warnings is a leak; an occasional warning is genuine concurrency, and the fix there is to raise `ScadaBridge:Transport:MaxConcurrentImportSessions` (each slot pins up to `MaxBundleSizeMb` of decrypted plaintext, so the cap is a memory budget).
### Conflict resolution and `BundleImportId` correlation
`PreviewAsync` compares each bundle artifact against the target environment using `ArtifactDiff`, classifying items as `Identical`, `Modified`, `New`, or `Blocker`. The operator assigns a `ResolutionAction` (`Add`, `Overwrite`, `Skip`, `Rename`) per item. `ApplyAsync` honours those resolutions in a single EF transaction and threads a new `BundleImportId` GUID through every per-entity audit row via the scoped `IAuditCorrelationContext`. This makes every configuration row written by a bundle import queryable as a group from the audit log.
@@ -122,10 +130,18 @@ Passphrase lockout operates at two levels: per-bundle (3-strike counter in `Bund
The two levels interact in a specific order, and it is load-bearing (register row 44 — the limiter had been registered, injected and unit-tested but never called, so only the per-bundle lockout was real):
1. **Per-bundle lockout first.** An already-locked bundle throws `BundleLockedException` without registering a rate-limiter attempt, so hammering a locked bundle cannot spend the client's hourly budget.
1. **Per-bundle lockout first.** An already-locked bundle throws `BundleLockedException` (Commons — register row 51 moved it out of `Transport.Import` for the same reason the throttle exception lives there: `ManagementService` does not reference `Transport`, so a lockout used to reach the CLI as `MapFault`'s generic "An internal error occurred" plus a correlation id) without registering a rate-limiter attempt, so hammering a locked bundle cannot spend the client's hourly budget.
2. **Per-IP throttle next, before the decrypt.** Over budget throws `BundleUnlockThrottledException` (Commons, so the management surface can catch it and return a curated message). Because it precedes the decrypt, a denial burns **no** per-bundle strike — otherwise the throttle would be a way to lock out a bundle someone is legitimately importing. Every unlock attempt is registered, successful ones included: the documented budget counts attempts, not failures.
3. Only then the AES-GCM decrypt, whose failure increments the per-bundle counter as before.
**The per-IP window is fleet-wide (register row 52).** It used to be a process-local `ConcurrentDictionary` on each central node, which had two consequences worth naming: the effective fleet cap was N × `MaxUnlockAttemptsPerIpPerHour` per client (a client that failed over to the other node got a fresh budget), and a node restart cleared the window outright. The window now lives in the central MS SQL `BundleUnlockAttempt` table — the HA store both central nodes already share, so this needs no new infrastructure, no new port and no consensus protocol. `BundleUnlockRateLimiter` keeps the policy; `SqlBundleUnlockAttemptWindow` is the store, resolving a scoped `IBundleUnlockAttemptRepository` from its own scope per call (it is a singleton, like the limiter). Three properties are load-bearing:
- **It fails OPEN.** A store error logs a Warning naming the client and the cap, and *permits* the attempt. This is a best-effort rate cap layered over a surface that is already authenticated and Administrator-gated, so the availability of legitimate imports outranks the cap — a database blip must not make the import wizard unusable. Falling back to the old process-local window instead was considered and rejected: it would still block a legitimate operator's 11th import during an outage, which is precisely the outcome fail-open exists to prevent. The Warning is the operator's signal that the fleet cap is not currently being enforced.
- **Accuracy is deliberately approximate.** The count-then-insert pair is not atomic across the two nodes, so genuinely simultaneous attempts can overshoot the cap by up to the number in flight. A conditional single-statement insert would close that at the cost of provider-specific raw SQL; the overshoot is bounded and self-corrects within the window.
- **Rows are pruned by two drivers sharing one cadence claim (register row 65).** The prune is claimed by CAS at most once per 15 minutes per process, bounded to 1000 rows per pass, retaining `2 ×` the window so a prune racing a count can never delete a row the count still needs, and wrapped in a try/catch that logs and swallows — retention maintenance must never fail the operator's actual unlock. This is the same shape `DeploymentService` uses for terminal deployment records. It originally rode the **unlock path only**, on the reasoning that Transport owned no central scheduler; the consequence was that a cluster which stopped importing entirely left its last rows in place indefinitely, since only a node actively serving unlocks ever pruned. Transport has since gained exactly such a scheduler — `BundleSessionEvictionService`, the central-only 1-minute session sweep — so it now also ticks `IBundleUnlockAttemptWindow.PruneAsync`, and the table converges whether or not anyone is importing. Because both drivers take the **same** claim, ticking on the 1-minute sweep costs at most one bounded DELETE per quarter hour and the two can never prune concurrently. `PruneAsync` is contractually non-throwing (unlike `TryRegisterAttemptAsync`, which throws and lets the limiter own the fail-open decision) — a background sweep has no caller to report to and must survive a database blip.
Behaviour the operator sees is otherwise unchanged: same config key, same semantics ("attempts, not failures"), same ordering relative to the per-bundle lockout, same `<unattributed>` bucket.
The client key is the caller's remote IP: the Central UI wizard reads it from `CircuitClientAddressProvider` (a scoped service that snapshots `Connection.RemoteIpAddress` at circuit establishment, since `HttpContext` is gone by the time a circuit submits a passphrase — the same pattern as `CookieAuthenticationStateProvider`), and the management endpoint stamps it on `ManagementEnvelope.ClientIp` for the CLI `transport preview` / `transport import` paths. A caller with no attributable address is counted against a single **shared** bucket rather than skipping the throttle — "unknown client" degrades to a stricter budget, never to none. As with the login throttle (`Component-Security.md`), per-client isolation is only real when ForwardedHeaders is configured for the fronting proxy; behind an untrusted proxy every caller collapses into the proxy's bucket.
```csharp
@@ -255,11 +271,11 @@ A template's scripts reference a `SharedScript` or `ExternalSystem` that exists
### Passphrase lockout
After 3 wrong passphrase attempts against the same bundle (keyed by `ContentHash`), `BundleImporter.LoadAsync` throws `BundleLockedException`. The session is unusable. Re-upload the bundle file to get a new session with a fresh counter. A `BundleImportUnlockFailed` audit row is written on each failed attempt.
After 3 wrong passphrase attempts against the same bundle (keyed by `ContentHash`), `BundleImporter.LoadAsync` throws `BundleLockedException`. The session is unusable. Re-upload the bundle file to get a new session with a fresh counter. A `BundleImportUnlockFailed` audit row is written on each failed attempt. Both the Central UI and the CLI surface the lockout reason verbatim — the exception lives in `Commons/Types/Transport/` so `ManagementActor` can curate it into a `ManagementCommandException` (register row 51).
### Unlock throttled ("Too many bundle unlock attempts")
`BundleUnlockThrottledException` — this **client** (not this bundle) has used its `MaxUnlockAttemptsPerIpPerHour` budget within the trailing hour. Re-uploading does not help; the budget is keyed by IP and rolls forward on a sliding window, and the counters are process-local (a central node restart clears them, deliberately, so an outage doesn't strand legitimate operators). The denial burns no per-bundle strike, so the bundle itself is still importable once the window frees up. Raise `ScadaBridge:Transport:MaxUnlockAttemptsPerIpPerHour` if a legitimate workflow (many encrypted bundles in one sitting) trips it routinely.
`BundleUnlockThrottledException` — this **client** (not this bundle) has used its `MaxUnlockAttemptsPerIpPerHour` budget within the trailing hour. Re-uploading does not help; the budget is keyed by IP, rolls forward on a sliding window, and is **fleet-wide and restart-durable** (register row 52 — see below), so failing over to the other central node does not help either. The denial burns no per-bundle strike, so the bundle itself is still importable once the window frees up. Raise `ScadaBridge:Transport:MaxUnlockAttemptsPerIpPerHour` if a legitimate workflow (many encrypted bundles in one sitting) trips it routinely.
### BundleImportAlarmScriptUnresolved / BundleImportCompositionUnresolved warnings
docs/deployment/topology-guide.md
+217 -9
View File
@@ -223,19 +223,35 @@ one of the ten replicated tables, and its retention DELETE is captured by CDC ex
ordinary write — by design, since LocalDb Phase 2 there is deliberately no purge-exemption path
(the same property that makes a mass DELETE dangerous, which is why `ReplaceAllAsync` was deleted
rather than reinstated). One oplog row is therefore queued per deleted event, and the backlog jumps
by the size of the day's expired batch.
by the size of the batch the tick deleted.
**The spike is bounded per tick** (register row 34, measured:
`docs/plans/2026-08-15-site-events-purge-burst-measurement.md`). One purge tick deletes at most
`ScadaBridge:SiteEventLog:MaxPurgedRowsPerTick` rows — default **250 000**, across the retention
purge and the cap trim combined — so the height of the spike is a property of the *configuration*,
not of how much history had expired. That matters because the burst used to be unbounded: a full
1 GB event log holds ~2.73 million rows, and expiring all of them at once queued 2 731 947 oplog
rows, **2.7x LocalDb's `MaxOplogRows` (default 1 000 000)** — enough for the maintenance tick to
prune the oplog and flag a *full ten-table snapshot resync*. A healthy peer blunts that (the oplog
drains while the purge runs — measured peak was 391 000, not 2.73 M), which is precisely the problem:
the old safety margin was a function of how fast the peer happened to be, and vanished when the peer
was down. Set `MaxPurgedRowsPerTick` to `0` to restore the old unbounded behaviour.
- **When.** Every `ScadaBridge:SiteEventLog:PurgeInterval` (default **24 h**), plus once at
startup. The timer is anchored to the **active node's process start**, not to a wall-clock hour,
so the burst lands at a different time of day after each failover or restart — do not expect it
at a fixed hour. The storage-cap trim (default 1 GB) can produce the same shape off-schedule, and
is usually the larger of the two.
startup — **except** after a tick that hit the ceiling, which schedules the next tick after
`ScadaBridge:SiteEventLog:PurgeCatchUpInterval` (default **5 minutes**) instead. So a large
backlog appears as a *run* of bounded spikes minutes apart rather than one huge one; a
worst-case 2.73 M-row backlog converges in ~11 ticks ≈ 55 minutes. The cadence is anchored to the
**end of the previous purge**, not to a wall-clock hour, so the burst lands at a different time of
day after each failover or restart — do not expect it at a fixed hour. The storage-cap trim
(default 1 GB) can produce the same shape off-schedule, and draws from the same per-tick budget.
- **Where it shows.** Only on a node with replication configured — the rig's site-a. site-b/site-c
have no capture triggers at all and report no backlog for a purge.
- **What healthy looks like.** `LocalDbReplicationConnected` stays **true** across the spike, and
the backlog drains back to ~0 as the peer acks the batch — within seconds to a couple of minutes
depending on batch size (delta messages are bounded by `LocalDb:Replication:MaxBatchBytes`,
default 2 MB, and secondarily by `MaxBatchSize`). No dead letters, no schema-mismatch errors.
the backlog drains back to ~0 as the peer acks the batch. Measured on a loopback pair at
~10 800 tombstones/second, so a full 250 000-row tick drains in well under a minute (delta
messages are bounded by `LocalDb:Replication:MaxBatchBytes`, default 2 MB, and secondarily by
`MaxBatchSize`). No dead letters, no schema-mismatch errors.
- **What is actually wrong.** `LocalDbReplicationConnected` **false** while the backlog climbs, a
backlog that keeps rising across successive readings rather than draining, or a backlog that
never returns near zero between bursts. Those point at the sync stream — an ApiKey mismatch
@@ -248,13 +264,69 @@ logs an Information line next to the purge count:
```
Purged 41230 events older than 30 days
Purged 41230 site_events rows on a replication-enabled node — a transient LocalDb oplog backlog
is expected while the deletes replicate to the peer. It drains on its own;
of up to that many rows is expected while the deletes replicate to the peer. It drains on its own;
LocalDbReplicationConnected staying true with the backlog returning to ~0 is the healthy signature.
A tick deletes at most 250000 rows, so the spike is bounded by that number rather than by how much
history had expired.
```
An unreplicated node logs only the first line. If a backlog spike has no such line near it in the
active node's log, the purge is *not* the explanation and the spike is worth investigating.
The second line is emitted **once per purge tick**, and its count covers the retention purge and
the storage-cap trim together. If you are comparing it against the `Purged N events older than…`
line above it, expect the two numbers to differ whenever the cap trim also ran — the note reports
the tick's whole contribution to the oplog, which is what a backlog spike is made of. (Through
2026-08-16 the note was emitted per phase, so a tick that ran both produced two near-identical
lines; register row 84.)
**A tick that hit the ceiling adds a Warning**, rate-limited to one line per 15 minutes so a
converging backlog does not flood the log:
```
Event log purge stopped at its per-tick ceiling of 250000 rows (250000 deleted); expired events
remain and the next purge runs in 00:05:00 instead of 1.00:00:00.
```
Every hit — throttled line or not — increments the counter
`scadabridge_site_event_log_purge_ceiling_hits`. **Read the counter's shape, not a single hit:** a
short run of hits that stops is a backlog converging (normal, e.g. the first purge after enabling
replication on a long-running site); hits that never stop mean the site writes events faster than
one paced tick can purge, i.e. `MaxPurgedRowsPerTick` or `PurgeCatchUpInterval` is mis-sized for
that site's event rate.
**Two dashboard surfaces make this reviewable rather than a log-archaeology exercise** (register
row 33):
- **The "Replication backlog" badge** on the site tile of the Health dashboard. It is deliberately
a *sustained* signal, not a bare threshold: the badge appears only once the backlog has stayed
above `ScadaBridge:HealthMonitoring:LocalDbBacklogThreshold` (default **1000**) continuously for
`ScadaBridge:HealthMonitoring:LocalDbBacklogSustainedFor` (default **15 minutes**). The daily
purge burst described above drains in seconds to a couple of minutes, so it never reaches the
dwell and never badges the site — which is the entire reason the dwell exists. **The per-tick
ceiling reinforces that rather than threatening it:** pacing lowers the spike (250 000 rows
maximum instead of an unbounded 2.7 M) at the cost of more ticks, and a 250 000-row tick was
measured draining in well under a minute — even allowing 10x for a real inter-node link that is
~4 minutes, against a 15-minute dwell and a 5-minute gap to the next tick. (The *unpaced* 2.73 M
burst drained in 6.0 minutes on loopback: inside the dwell, but with only 2.5x of margin, so a
materially slower link would have badged.) A bare threshold
would fire once a day on every replicated site and be ignored within a week. The dwell is
validated to be at least one `ReportInterval`, since a shorter window could be satisfied by a
single sample and would collapse back into a bare threshold.
- **The "LocalDb Replication Backlog" trend chart** in the dashboard's per-site Site Health Trends
panel, backed by the KPI history store (sampled every minute, retained 90 days raw / 365 days
hourly). This is what answers the question a point-in-time reading cannot: *did that spike
drain?* A past purge burst appears as a spike that returns to ~0; a real fault appears as a
level that never comes back down.
**Both surfaces are silent on an unreplicated site, by design.** `LocalDbOplogBacklog` and
`LocalDbReplicationConnected` are nullable, and null means "no reading", never zero. The KPI
sampler therefore writes **no sample at all** rather than a fabricated 0, so site-b/site-c show an
empty chart rather than a flat zero line, and they never accumulate the dwell anchor the badge
keys off. A gap in those series means "not replicated / not reporting" — which is the truth. Note
that `false` and null differ for the connected flag: a configured link that is DOWN samples as 0,
while no link at all samples as nothing.
### Site Pair Upgrades — stop and start BOTH nodes together
**A rolling upgrade of a site pair, one node at a time, is no longer supported.** It worked while
@@ -265,6 +337,18 @@ will run, but they will not converge, and the divergence is silent.
Stop both nodes of a site pair, upgrade both, then start both.
**A release that adds a column to a replicated table is the same constraint, enforced louder.**
`ReplicatedTable.ComputeDigest` hashes every column name and declared type, and the sync handshake
compares those digests fail-closed — so a mixed-version pair whose builds disagree about a table's
columns does not diverge silently, it refuses to sync with a `LocalDbSchemaMismatchException`.
Stopping and starting both nodes together, which is already the rule above, is the whole fix; no
manual trigger or oplog migration is needed, because `RegisterReplicated` re-reads
`PRAGMA table_info` on every boot and rebuilds the capture triggers (DROP + CREATE) over the
current column set. The site schema's own `AddColumnIfMissing` runs before registration on that
same boot, so a node picks up the new columns and the matching triggers in one start.
(Most recent instance: `external_systems` gained `max_retries` / `retry_delay_ms` for the
per-system store-and-forward retry policy.)
**Related bound — do not leave one node of a pair offline for long.** A node absent for longer than
`LocalDb:Replication:TombstoneRetention` (default **7 days**) can **resurrect deleted rows** when
it rejoins: deletes replicate as HLC-ordered tombstones, and once a tombstone is pruned there is
@@ -273,6 +357,103 @@ a rejoin is safe and self-correcting (verified live: a node stopped and restarte
with both nodes byte-identical and zero duplicates). Beyond it, rebuild the returning node's
database from its peer rather than letting it rejoin.
### Docker-rig data files on macOS/OrbStack — a virtiofs corruption + availability hazard
**Scope: the local `docker/` and `docker-env2/` development rigs on macOS only.** This is a
host-level property of the filesystem the rig's data files sit on, **not a ScadaBridge defect** and
not something a production Linux deployment is exposed to. It is documented here because it has
already cost two databases and half a session, and because the recovery is non-obvious.
Every ScadaBridge data file on the rig rides a **relative bind mount** out of the repo working tree
— `./site-a-node-a/data:/app/data` and its siblings, which is where `site-localdb.db`,
`auditlog.db` and the secrets store all live. On macOS that path crosses **virtiofs** to reach the
Linux VM, and **virtiofs does not give SQLite true `fsync` durability**. SQLite's crash-safety
contract assumes a successful `fsync` means the bytes are on stable storage; when that is not true,
an abrupt container kill mid-write can leave a torn file. `docker compose up --force-recreate`
— which `deploy.sh` uses — is exactly such an abrupt kill.
Two distinct failure modes have been observed live (both 2026-08-15):
**1. Silent database corruption.** Both site-a LocalDb files were found malformed:
```
SQLite Error 11: 'database disk image is malformed'
```
Note how differently the two nodes presented, because this is the part that makes it hard to spot:
- **site-a-b** surfaced it as a *running* fault — every sync call threw, replication reported
`connected: false` with a backlog of 892, and the faulted session retried about once a minute.
The node stayed up and otherwise healthy-looking.
- **site-a-a** was corrupt too, but **latently** — nothing on the running build ever read the
damaged pages. It only became fatal on the *next upgrade*, when a new build's boot-time
re-registration read pages the old build never touched: `SqliteLocalDb` construction threw and
the container **SIGABRT crash-looped** (×12).
The lesson to carry: a clean-looking node proves nothing about its file. Corruption can sit latent
until an unrelated code change happens to read further into the database.
**2. The OrbStack VM idle-sleeping mid-operation.** The VM can suspend under its idle policy while
work is in flight. `~/.orbstack/log/vmgr.log` records it plainly:
```
vmgr ... msg=sleep
```
On wake, the virtiofs layer logs repeated errors of the form:
```
refresh_nodeid: race with new lookup
```
and `docker` hangs — commands stop returning, `docker ps` comes back empty — for as long as it
takes to notice. **The fix is `orb restart docker`**; data survives and containers auto-restart.
Observed cost: roughly 20 minutes of a session lost to diagnosing what looked like a cluster
failure but was a host one. If the rig goes unresponsive as a whole, check `vmgr.log` **before**
investigating ScadaBridge.
#### Recovery runbook — LocalDb corruption on a site pair (proven)
This is the procedure that recovered site-a. It relies on the pair being able to reseed itself,
so it is a *replicated* pair's recovery; an unreplicated node has no peer to reseed from and must
be restored from a backup instead.
1. **Stop BOTH nodes of the site pair, together.** Not one at a time — this is the same
stop-and-start-together rule as *Site Pair Upgrades* above, for the same reason.
2. **Quarantine the database and its sidecars — do not delete them.** Rename all three, keeping
them for forensics:
```sh
# per affected node's data directory
mv site-localdb.db site-localdb.db.corrupt-$(date +%F)
mv site-localdb.db-wal site-localdb.db-wal.corrupt-$(date +%F)
mv site-localdb.db-shm site-localdb.db-shm.corrupt-$(date +%F)
```
The `-wal` and `-shm` files **must** move with the main file. Leaving a stale WAL beside a fresh
database is its own corruption.
3. **Start both nodes together.** The fresh files reseed themselves from two independent paths:
`SiteReconciliationActor`'s node-startup fetch pulls config back from central when central
reports gaps, and LocalDb's **CDC snapshot resync** merges the peer's rows under LWW.
4. **Verify convergence**, do not assume it: `LocalDbReplicationConnected` should read `true` and
`LocalDbOplogBacklog` should settle to ~0 on the site health report. (See *Reading the
replication backlog* above for why a transient spike is normal and not a fault.)
On the live recovery, both quarantined files were large (394 MB and 416 MB), the pair converged to
`connected: true` / backlog 0, and a schema change in flight at the time additionally proved the
digest handshake green.
#### Mitigation options, with their trade-offs
None of these is applied today; the rig deliberately still runs on bind mounts, and the accepted
position is the runbook above. Recorded so the choice is a choice:
| Option | Gains | Costs |
|--------|-------|-------|
| Move rig data dirs to **named docker volumes** | Files live on native ext4 inside the VM, out of virtiofs entirely — removes the corruption mechanism at the source | Loses the host-side inspectability the debugging sessions lean on constantly: no more opening `site-localdb.db` from the Mac, no `ls` of a node's data dir from the repo tree. Requires a compose change. |
| **Disable OrbStack's idle pause** | Removes failure mode 2 (the mid-operation sleep and the virtiofs `refresh_nodeid` storm) | A host-level setting outside this repo, so it cannot be version-controlled or applied by `deploy.sh` — it silently does not exist on a new machine. Does nothing about failure mode 1. |
| **Accept, and use this runbook** (current) | No changes to compose or host config; full host-side inspectability retained | Corruption recurs on the next unlucky `--force-recreate`, and can sit latent until a later upgrade turns it into a crash loop. |
### Central-Site Communication
Three transports cross the boundary, not one — **all now gRPC or HTTP; Akka ClusterClient was removed
@@ -339,6 +520,33 @@ hardening and needs no change to the key design.
| Site node | 2 cores | 4 GB | 20 GB | SQLite databases grow with S&F |
| SQL Server | 4 cores | 16 GB | 100 GB | Shared across central cluster |
#### Sizing a site's data volume — `MaxStorageMb` is not the whole story
`ScadaBridge:SiteEventLog:MaxStorageMb` (default **1 GB**) bounds **`site_events` and its four
indexes**, and nothing else. It is deliberately *not* a cap on the consolidated LocalDb file
(register row 81 — decision record
`docs/plans/2026-08-17-site-event-log-cap-semantics-decision.md`). Before that change it did
measure the whole file, which on a replicated node made the cap unenforceable: the trim's own CDC
oplog rows and tombstones grew the file as fast as the deletes shrank it, and the trim stopped
2.5x over cap. Tying the event log's budget to the peer's liveness was the defect, not a feature
worth preserving.
So budget the volume as the **sum** of:
- **`MaxStorageMb`** — the event log proper.
- **LocalDb replication bookkeeping** — `__localdb_oplog` and `__localdb_row_version`, bounded
separately by `LocalDb:Replication:MaxOplogRows` (default 1 000 000) and `TombstoneRetention`
(default 7 days). This is transient: it drains as the peer acks and as tombstones expire, but it
is not small during a burst. Measured worst case (register row 34): purging a full 1 GB event log
left **707 MB** of bookkeeping in the file. Unreplicated sites carry none of this — they have no
capture triggers at all.
- **The other eight consolidated tables** plus the separate, unreplicated audit database at
`AuditLog:SiteWriter:DatabasePath`.
The 20 GB figure above has ample headroom for all three at default settings. Raising
`MaxStorageMb` on a replicated site should be paired with a look at the oplog settings, not done
in isolation.
### Network Bandwidth
- Health reports: ~1 KB per site per 30 seconds = negligible
docs/plans/2026-07-08-deferred-work-register.md
+61 -18
View File
@@ -20,33 +20,76 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round
| 24 | M8 large-bundle performance hardening | `docs/plans/2026-06-15-stillpending-completion-design.md:106` — "Small follow-ups logged (not blocking): … large-bundle/perf hardening" | Logged as a non-blocking follow-up when M8 shipped and never given an artifact: **no plan, no task entry, no perf/load test exists** (`tests/…Transport.Tests/Import/BundleImporterLoadTests.cs` is a `LoadAsync` unit suite despite the name). No measured problem; the only sizing controls in place are the 5-minute CLI transport timeout, `LineDiffer`'s `MaxInputLines`=4000 summary-only cap, and `MaxConcurrentImportSessions`=8. | First real bundle that times out, exhausts memory, or makes the import wizard's diff step unusable |
| 25 | ~~**Phase-8 WP-4 target-scale load test** (10 sites x 500 instances x 75 tags = 37,500 subscriptions/site, 375,000 total)~~ **RESOLVED 2026-08-15 — run, measured, evidenced.** The 107-byte stub is replaced by a real full-scale run: **10 x 500 x 75 = 375,000 live tag subscriptions**, 37,518 tag updates/s achieved against 37,500 nominal (100.0%), **45,021,375 updates offered** over a 20-minute steady-state window on an M4 Pro / 14-core / 48 GB host with the 8-node `docker/` rig still running. Headline numbers: tag update latency **P50 0.88 ms / P99 4.57 ms / max 37.41 ms** end-to-end (DCL boundary to stream subscriber, the emit instant carried verbatim through production code); **0 events dropped** at 100 live subscribers; health report collect+ingest **P99 0.31 ms** with 10/10 sites tracked; debug view snapshot under load **P99 2.19 ms, 0 timeouts**; 500 instances deployed to a site in **2.6 s**; CPU **2.9% of the box**; working-set slope **+8.83 MB/min**. **11 clean passes, 1 pass with a caveat (F1: 20 min with zero gen-2 collections cannot fully settle the leak question — the 1-hour run would), 0 failures.** Harness: `tests/ZB.MOM.WW.ScadaBridge.LoadHarness/` (standalone Exe — the `Category=Performance` trait enables a filter but does NOT exclude by default, so a 20-minute test could not live in `PerformanceTests`) plus a CI-scale smoke `[Fact]` (`PerformanceTests/TargetScale/TargetScaleHarnessSmokeTests.cs`). Real: per-site ActorSystem + LocalDb, the real DCL via the documented `DataConnectionFactory.RegisterAdapter` seam, real `InstanceActor`s, real `SiteStreamManager`, real `StreamRelayActor` + production-capacity bounded channel, real S&F, real health collector/aggregator; only the socket hops are stood in for. **Results: `docs/plans/2026-08-15-target-scale-load-test-results.md`; design + deviations + thresholds: `docs/plans/2026-08-15-target-scale-load-test-design.md`.** | as before | Closed. Residual scope: `[xc-3]` site-event-log retention volume, `[xc-4]` audit-vs-central degradation, `[xc-5]` large-template flattening and `[xc-8]` UI responsiveness are NOT covered by this harness (central-cluster/browser concerns) and remain open WP-4 scope — see results doc SS6. | Closed 2026-08-15. |
| 26 | Ipsen MES MoveIn tail: leak-test (`-LT`) receivers + routing, PLC-output-flag writes, `Z28062` BTDB data completeness | `docs/plans/2026-06-16-ipsen-mes-movein.md:409` ("Out of scope (future)"); design `2026-06-16-ipsen-mes-movein-design.md:58-60, :196-198` | Customer-site scope, not a platform gap. `-LT` routing needs an MES-receiver child + Galaxy reference that do not exist on the reactor template (any `-LT`/unknown suffix returns `WasSuccessful=false` with an "unsupported side/target" message by decision); `MoveInComplete`/`Successful`/`ErrorText` are **PLC-owned** by locked decision, so ScadaBridge deliberately does not write them; `Z28062` completeness is an operational data fix, not code. Note the separate alarm-status path already handles the suffix — `_LT` is stripped before side-scoping (`2026-06-30-mes-alarm-status-api.md:158`). | Ipsen creates the leak-test receiver + Galaxy reference, or asks ScadaBridge to own the PLC-output flags — otherwise a **candidate won't-do** (`[PERM]`) at the next Ipsen scope review |
| 27 | External-system per-system retry config (`MaxRetries`/`RetryDelay`) never reaches sites, and has no CLI/management surface | Found live 2026-08-01 (rig session, #11 gRPC live checks) | Two stacked gaps: (a) `ExternalSystemArtifact` (Commons) carries `TimeoutSeconds` but NOT `MaxRetries`/`RetryDelay`, and the site `external_systems` table has no such columns — so a centrally-configured retry policy is silently ignored on sites; every cached call buffers with the S&F default (`DefaultMaxRetries` 50 × `DefaultRetryInterval` 30s ≈ 25 min to park). (b) `Create/UpdateExternalSystemCommand` don't expose the fields either — the only way to set them today is a direct DB edit of `ExternalSystemDefinitions`. Transport bundles DO carry them (arch-review 05 "ES retry config"), which masks the gap in export/import round-trips. Fix is additive: extend the artifact + site schema + apply path, and add `--max-retries`/`--retry-delay` to the CLI. | First operator who tunes retry policy on an external system and expects site cached calls to honor it |
| 28 | Health-dashboard "Trigger failover" confirm dialog's confirm button is labeled **"Delete"** | Found live 2026-08-01 (rig session, #11 TriggerSiteFailover check) | The DialogService confirmation host's default destructive-action label leaks through — the dialog copy is correct but the red confirm button says "Delete" for a failover. One-line fix: pass an explicit confirm label ("Fail over") at the Health-dashboard call site (and audit other confirm-dialog call sites for the same default). | Next Central UI session |
| 29 | **1-hour sustained run to settle the memory-slope ambiguity (load-test finding F1)** | `docs/plans/2026-08-15-target-scale-load-test-results.md` F1 (Low) | The row-25 20-minute run PASSED the working-set threshold (+8.83 MB/min < 20) but recorded **zero gen-2 collections across 45M events**, so the +19.52 MB/min managed-heap drift on an uncompacted gen-2 is ambiguous — the heap demonstrably sawtooths, but 20 min cannot distinguish "gen-2 not yet collected" from a slow leak. Not a defect finding; an evidence gap. The harness exists (`tests/ZB.MOM.WW.ScadaBridge.LoadHarness/`) and the full 1-hour protocol is one documented command (results doc §7) — deferred only because the measured window was scaled for practicality. | Before production go-live at target scale, or first field report of unexplained site-node memory growth |
| 27 | ~~External-system per-system retry config (`MaxRetries`/`RetryDelay`) never reaches sites, and has no CLI/management surface~~ **RESOLVED 2026-08-15 (`41a5ecf1`).** The gap was transport + persistence only — the consumption end already worked (`ExternalSystemClient.CachedCallAsync` already passed `system.MaxRetries`/`RetryDelay` to `EnqueueAsync` with 0-as-unset, and the Central UI form already collected both fields, inert until now with defaults 3×5s that will take effect on next deploy). Shipped: artifact fields, additive `site_command.proto` fields 7/8 (regen via `docker/regen-proto.sh`, csproj left clean), site `external_systems` columns `max_retries`/`retry_delay_ms` as REAL COLUMNS — not a JSON bag; verified safe against the AckTime precedent in the LocalDb source (capture triggers are DROP-then-CREATE from `PRAGMA table_info` each boot, and the fail-closed handshake digest turns a mixed-version pair into a loud sync refusal, no new constraint since both-nodes-together is already the standing pair rule; `timeout_seconds` set the additive-column precedent on this exact table) — apply path, and CLI `--max-retries`/`--retry-delay-seconds` (+`--retry-delay` alias). Bonus fix: `ArtifactDiff.CompareExternalSystem` never compared the retry fields, so a bundle import changing a retry budget previewed as "Identical". Fail-before verified on the site round-trip + apply tests; 12 suites green. | Found live 2026-08-01 (rig session, #11 gRPC live checks) | Closed 2026-08-15. | — |
| 28 | ~~Health-dashboard "Trigger failover" confirm dialog's confirm button is labeled **"Delete"**~~ **RESOLVED 2026-08-15 (`0db253b5`).** Structural, not a missing argument: `IDialogService.ConfirmAsync` had NO label parameter — `DialogHost` derived the verb from `danger` alone (`danger ? "Delete" : "Confirm"`), so no call site could ever have passed one. New optional `confirmLabel`; `danger` now selects styling only. Full audit of all 34 `ConfirmAsync` call sites (the only confirmation path — no `window.confirm`, no per-page modals): NINE were wrongly labeled "Delete" — both failover buttons, secured-write **Approve** and **Reject**, instance Disable, and four Discard actions — all now name their action; 18 genuine deletes + 7 non-danger sites verified correct. The `danger→"Delete"` fallback deliberately kept: forcing an explicit label would churn ~20 correct sites, and a missed site would silently degrade to "Confirm", detectable only by the rig-dependent Playwright suite — contract pinned in the interface XML doc, Component-CentralUI.md, and a fallback test instead. Gotcha recorded in the tests: adding an optional parameter silently un-matches existing 3-arg NSubstitute stubs (returns null Task → NRE). Follow-ups: rows 75 (Approve's red styling), and the bUnit real-dialog deadlock hazard documented in `FailoverConfirmLabelTests`. | Found live 2026-08-01 (rig session, #11 TriggerSiteFailover check) | Closed 2026-08-15. | — |
| 29 | ~~**1-hour sustained run to settle the memory-slope ambiguity (load-test finding F1)**~~ **RESOLVED 2026-08-15 — run, and F1 is settled: NOT a leak.** Full 1-hour protocol executed (same harness/scale, `--sustain-minutes 60`, 135,041,250 updates at 37,511/s = 100.0 % of nominal, concurrent with the live docker rig AND a central failover drill on the same host). Decisive: **working-set slope −28.89 MB/min** (2,722 → 1,757 MB over the 3,600 s window, peak 3,258 MB) with **one gen-2 collection observed** and threads flat at 59 — a working set that FALLS ~1 GB across a sustained hour is incompatible with a slow leak; the positive managed-heap least-squares slope was the sawtooth-segment artifact F1 hypothesised. All other metrics held or improved vs the 20-min run: P99 4.01 ms, 0 healthy-subscriber drops, 744/744 debug snapshots, S&F drain to depth 0. Full numbers appended to the results doc F1 section. | `docs/plans/2026-08-15-target-scale-load-test-results.md` F1 postscript | Closed 2026-08-15. | — |
| 30 | **WP-4 residual scope: `[xc-3]` site-event-log retention volume, `[xc-4]` audit-load-vs-central degradation, `[xc-5]` large-template flattening time, `[xc-8]` UI responsiveness under load** | `docs/plans/2026-08-15-target-scale-load-test-results.md` §6 + row 25 closure note | The row-25 harness is a site-runtime rig; these four WP-4 sub-criteria are central-cluster/browser concerns it deliberately does not cover. Recorded so the row-25 closure never reads as "WP-4 fully done" — phase-8 WP-4 is closed for the site-runtime data path only. | Before declaring phase-8 WP-4 complete, or production go-live at target scale |
| 31 | **Unpaced-burst absorption is bounded by the shared publish stage, not per-subscriber buffers (load-test finding F3 caveat)** | `docs/plans/2026-08-15-target-scale-load-test-results.md` F3 | Measurement-backed design note, no defect: slow-subscriber isolation is structural and total (a stalled reader loses events only in its own bounded `DropOldest` channel; healthy subscribers delivered 100.00%), but the publish `Source.ActorRef` **upstream** of the `BroadcastHub` is shared, so an unpaced burst degrades every subscriber equally — that stage, not per-subscriber buffering, is the burst ceiling. | First field report of event loss on *healthy* subscribers during a burst, or any redesign of the site stream publish stage |
| 32 | **`QueueDepthGaugeTests` full-suite flake — shared static gauge (load-test finding F4)** | `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/QueueDepthGaugeTests.cs`; surfaced by the row-25 verification runs (`docs/plans/2026-08-15-target-scale-load-test-results.md` F4) | Pre-existing, test-only: fails when the full suite runs concurrently, passes in isolation — the gauge under test is process-global static state shared across test classes. Zero `src/` involvement; needs test isolation (per-test meter/collector instance), not a product fix. | Next test-infra session, or the first time it fails a CI/full-suite run that matters |
| 33 | **`LocalDbOplogBacklog` has no threshold, KPI history, or dashboard treatment** | Found by R7 (`docs/plans/2026-08-15-residual-remediation-plan.md`) while documenting the purge burst | `SiteAuditBacklog` drives a configurable threshold warning on the site tile, but the LocalDb backlog is reported and Prometheus-exported with nothing consuming it for alerting or trending — it is not among the metrics `SiteHealthKpiSampleSource` samples into KPI history, so the purge-burst spike R7 documents cannot be reviewed after the fact; an operator must be watching live. Candidate: add it to the KPI sample source and/or a sustained-backlog threshold badge. Pairs naturally with the R1 per-table-snapshot work, which changes snapshot/backlog behavior. | Next health-dashboard session, or first operator question about a backlog spike they didn't see live |
| 34 | **`site_events` purge burst is unbounded in the oplog and unmeasured at scale** | Found by R7 | Retention deletes are sliced at 1000 rows/statement, but the whole expired set is deleted in one tick, so a replicated site's oplog gains one row per deleted event with no pacing — a first purge after enabling replication (or after a long low-purge period) queues the entire batch at once. The wire path is bounded (`MaxBatchBytes`), but the oplog write itself is not paced, and no measurement exists for batch size or drain time at target scale. Candidate: measure on the rig or in the load harness; consider a per-tick delete ceiling if drain time is material. | Before enabling replication on a site with a large existing `site_events` backlog, or the next load-harness session |
| 35 | **`DeploymentRecords` index does not cover the deployment-list sort key** | Found by R3 (`35ce1413`) | The table has a single-column `IX DeployedAt`; the new list query orders `DeployedAt DESC, Id DESC` and joins `Instance`, so SQL Server sorts per page. A composite `(DeployedAt DESC, Id DESC)` index would serve the ordering directly — deliberately not done in R3 because it needs an EF migration and the terminal-record purge bounds the table. *(Low)* | Deployment list measurably slow at a real fleet's retention depth |
| 36 | **Deployments free-text search is unindexable by construction** | Found by R3 | Four `LIKE '%term%'` predicates (instance name, deployment id, revision hash, user) — leading wildcards preclude index seeks, so a search is a scan of the join. Acceptable at the purge-bounded row count. Fix if hot: anchor the id/hash legs as prefix matches, or full-text. *(Low)* | Search latency complaints on the Deployments page |
| 37 | **Two paged reads of `DeploymentRecords` with different contracts** | Found by R3 | `QueryDeploymentSummariesAsync` (CLI/ManagementActor: offset, no total, no instance join, instance-id-list scoping) and R3's `QueryDeploymentListPageAsync` (UI: total + instance join + site-id scoping) duplicate query logic. Consolidating on one contract — giving the CLI a total, both scoping by site — removes the duplication. Genuinely different contracts today, so not urgent. *(Low)* | Next time either query's shape changes |
| 32 | ~~**`QueueDepthGaugeTests` full-suite flake — shared static gauge (load-test finding F4)**~~ **RESOLVED 2026-08-15 (`65b19516`).** Reproduced first (2 failures in 5 parallel runs — the gauge read a sibling class's provider). The register's suggested fix (per-test meter) was rejected on inspection: the slot is process-global by product design and 3 of the 6 facts assert the slot-ownership semantics themselves (compare-and-clear identity, stop/restart re-seed), so per-test instances would change what they prove. Correct isolation is mutual exclusion — the three `StartAsync`-calling classes now share xunit collection `QueueDepthGaugeCollection` (with a `[CallerFilePath]`-anchored source-scan drift guard that names an offending file), the other eight classes stay parallel, and `parallelizeTestCollections` was flipped back ON so the fix is exercised rather than masked. Negative control: dropping the attribute fails the guard AND resurfaces the original flake in the same run. 8/8 green parallel runs. | `tests/…StoreAndForward.Tests/QueueDepthGaugeTests.cs`; surfaced by the row-25 verification runs | Closed 2026-08-15. | — |
| 33 | ~~**`LocalDbOplogBacklog` has no threshold, KPI history, or dashboard treatment**~~ **RESOLVED 2026-08-15 (`8da60f57`).** Both halves shipped: sampled into KPI history via `SiteHealthKpiSampleSource` (conditionally — 12 samples unreplicated / 14 replicated, so a series gap truthfully means "not replicated") plus a backlog trend chart on the site health panel, and a SUSTAINED threshold badge (`LocalDbBacklogThreshold` 1000 held for `LocalDbBacklogSustainedFor` 15 min, anchored by `LocalDbBacklogAboveSince` in the aggregator) — a bare instantaneous threshold would fire on every replicated site every day from the documented purge burst. Null backlog is a deliberate third branch: it leaves the anchor unchanged (neither evidence of a drain nor of a backlog), so unreplicated sites can never anchor and can never be badged. NOTE the row's premise was false — the `SiteAuditBacklog` "configurable threshold badge" it told us to mirror was described twice in `Component-HealthMonitoring.md` but NEVER BUILT; both stale doc claims corrected, and the new badge was modeled on the real Metrics-stale badge instead. | Found by R7 while documenting the purge burst | Closed 2026-08-15. | — |
| 34 | ~~**`site_events` purge burst is unbounded in the oplog and unmeasured at scale**~~ **RESOLVED 2026-08-16 (`d9a35846`) — measured, and paced.** Measured with the REAL `EventLogPurgeService` over a real `RegisterReplicated` LocalDb file + a loopback sync pair at the rig's `MaxBatchSize: 64`: 393 B/row incl. indexes ⇒ a 1 GB event log holds **~2.73 M rows**, and expiring them in one tick took 285 s and queued **2 731 947 oplog rows = 2.7x LocalDb's `MaxOplogRows` default (1 M)** — enough to make the maintenance tick prune the oplog and flag a FULL ten-table snapshot resync, bought by a routine retention purge. The decisive nuance: the *resident* backlog is produced−minus−drained, so a healthy peer blunts it (391 k peak on loopback, drained 6.0 min) — the old safety margin was a function of peer throughput and vanished with the peer down. Fix: `MaxPurgedRowsPerTick` (default **250 000**, `0` = unlimited escape hatch), one budget SHARED across retention + cap trim; a ceiling-limited tick reschedules after `PurgeCatchUpInterval` (**5 min**, validated `> 0` and `<= PurgeInterval`) so worst-case retention converges in ~55 min instead of 11 days; purge loop is now a delay loop anchored to the END of the previous purge; rate-limited Warning + counter `scadabridge.site.event_log.purge_ceiling_hits`. No purge-exemption path added. Row-33 interplay verified empirically (500 k drained in 46 s vs the 15-min dwell). Fail-before: `PurgeBudget` neutralised → 6 of the new tests fail, all 19 pre-existing pass. Measurement + decision record: `docs/plans/2026-08-15-site-events-purge-burst-measurement.md`. Side-finding recorded as row 81 (storage cap unenforceable on a replicated node). | Found by R7 | Closed 2026-08-16. | — |
| 35 | ~~**`DeploymentRecords` index does not cover the deployment-list sort key**~~ **RESOLVED 2026-08-15 (`be29bf47`).** Migration `20260815165452_AddDeploymentRecordListSortIndex` + hand-written `docs/plans/sql/AddDeploymentRecordListSortIndex.sql`: composite `IX_DeploymentRecords_DeployedAt_Id (DeployedAt DESC, Id DESC)`; the old single-column index DROPPED as strictly redundant (`Id` is the clustered PK and was already in the old index's leaf as the row locator — same leaf width, now a declared ordered key; keeping it would cost a second index on every insert/status transition for no read benefit). Serves the three reads sharing the key (`QueryDeploymentListPageAsync`, `QueryDeploymentSummariesAsync`, `GetCurrentDeploymentStatusAsync`). `Id` in the key is also a CORRECTNESS matter: a bulk site deploy stamps many rows in one `DeployedAt` instant, so without the tie-break keyset pages can repeat or drop rows. Recorded caveat: the optimizer may already have served the ordering via a backward scan of the old index, so the "sorts per page" premise was possibly optimizer-dependent — the composite makes it deterministic and self-documenting; an actual plan capture against the rig would quantify the before/after. The ConfigurationDatabase suite exercises the descending composite as real DDL via SQLite `EnsureCreated()`, proving provider portability. | Found by R3 (`35ce1413`) | Closed 2026-08-15. | — |
| 36 | **Deployments free-text search is unindexable by construction** | Found by R3; re-verified 2026-08-17 (Low-tail sweep) | Four `LIKE '%term%'` predicates (instance name, deployment id, revision hash, user) — leading wildcards preclude index seeks, so a search is a scan of the join (`DeploymentManagerRepository.cs:100-104`; only searched column with any index is `DeploymentId`, unseekable under a leading wildcard). Acceptable at the purge-bounded row count. **2026-08-17 refresh: the row undercounted the cost by half** — `BuildListQuery` is shared with `GetDeploymentStatusCountsAsync`, so one search = TWO scans of the join (list + status tiles), mitigated by the UI's trailing-edge debounce (one pair per pause, not per keystroke). Anchoring the id/hash legs was evaluated and REJECTED as a micro-fix: `DeploymentListFilter.Search`'s doc comment promises substring matching and a 4-case `[Theory]` deliberately pins mid-string matches — changing it is a contract renegotiation, not an optimization. Fix if hot: renegotiate to prefix on id/hash legs, or full-text. *(Low)* | Search latency complaints on the Deployments page |
| 37 | **Two paged reads of `DeploymentRecords` with different contracts** | Found by R3; re-verified 2026-08-17 (Low-tail sweep: both shapes intact, `DeploymentManagerRepository.cs:110-149` and `:399-449`) | `QueryDeploymentSummariesAsync` (CLI/ManagementActor: offset, no total, no instance join, instance-id-list scoping) and R3's `QueryDeploymentListPageAsync` (UI: total + instance join + site-id scoping) duplicate query logic. Consolidating on one contract — giving the CLI a total, both scoping by site — removes the duplication. **2026-08-17: the duplication has now DRIFTED into real behavioral divergences, recorded as rows 86–88 (site-scope authorization divergence, CLI unbounded read + non-transactional scope resolution, silent `--status` parse fallback) — a consolidation should resolve those rows with it.** *(Low structurally; the drift itself is row 86, Medium)* | Rows 86–88, or next time either query's shape changes |
| 38 | **OtOpcUa: `RoslynVirtualTagEvaluatorTests` racing-clear test is load-fragile** (OtOpcUa repo, not this one) | Found by R4 during the full-suite bump verification | `Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed` fails under full-solution parallel test load, passes 3/3 isolated: its design assumes at most one `ClearCompiledScripts` lands inside one evaluation (so the product's one-shot retry suffices); under contention two can. Same timing-assumption family as the R5 sweep, but lives in OtOpcUa. Test-side fix: bound clear cadence relative to observed evaluation latency, or assert on retry-exhaustion instead of zero failures. | Next OtOpcUa test session, or when it fails a run that matters |
| 39 | **OtOpcUa: late-enabled replication cannot baseline pre-existing rows** (OtOpcUa repo, not this one) | Found by R4; documented in OtOpcUa's CLAUDE.md | `LocalDbSetup.OnReady` registers all four tables unconditionally on every driver node, so it can never pass `baselineExistingRows: true` (only correct when gated on replication being configured). Turning replication ON for a long-running default-OFF node converges on subsequent writes only — existing rows are never baselined. ScadaBridge closed this class with conditional registration + boot-time `DeregisterReplicated` self-heal (WP1.3/WP3.3); porting it is a behavior change beyond a pin bump (registered-set pinned by tests, deregistration must be pair-symmetric). Low current impact: site-a was enabled from fresh. | Before enabling replication on an established OtOpcUa node |
| 40 | **Wall-clock sleeps used as synchronization in tests (absence-assertion class)** | Found by the R5 sweep (`9fb52153`) — a different class from the one swept, deliberately left unfixed | Tests that sleep, then assert an *absence*: `ScriptDeadlineAtEnqueueTests.cs:105,128` (two `Thread.Sleep(2s)`, with `Assert.Equal(0, scheduler.DetachedThreadCount)` depending on the sleep out-racing the watchdog — worst offender), `DeploymentWarmThenGateTests.cs:117`, `SiteAlarmLiveCacheServiceTests.cs:195,216`, `DataConnectionActorTests.cs:435` (`Task.Delay(200)` to "settle" after an `AwaitCondition`). These fail toward false-pass rather than flake, so they don't trip CI — but each is a timing pin that weakens what the test proves. Fix pattern: replace the sleep with an explicit observable edge (or assert on the positive event with a bounded wait plus a negative control). | Next test-infra session, or the first time one of these areas regresses without its test catching it |
| 40 | ~~**Wall-clock sleeps used as synchronization in tests (absence-assertion class)**~~ **RESOLVED 2026-08-15 (`7a7d345b`).** All six listed sites replaced with observable edges + positive controls, strengthening proven by stash-based negative controls: `ScriptDeadlineAtEnqueueTests:128` now pins the watchdog via a genuinely-wedged positive control (armed strictly later, self-releasing) + per-script absence assertion + detach counter pinned at 1 — **disabling the entire stuck-script watchdog passed the OLD test 2/2 and fails the new one**; `:105` now proves the run was still queued when its budget expired via `QueueDepth` edges (the actual design claim — the old sleep asserted nothing about it); `DeploymentWarmThenGateTests:117` (+2 sibling `Task.Delay(500)`s) — the empty-site zero is now the manager's own observation (phantom-startup control: old passed 2/2, new fails); `SiteAlarmLiveCacheServiceTests:195,216` poll a new subscription-cancellation observable across 3 linger windows, each closing with a positive control (control: linger cancellation disabled + viewer guard dropped — `Resubscribe_Within_Linger` old passed 2/2, new fails; honest caveat: the other case's old later assertion did already catch that break, so its gain is the observable + control, not a false-pass fix); `DataConnectionActorTests:435` waits for the Connected health publication so the subscribe provably runs against Connected (the sleep guarded no assertion — determinism gain only). Bonus: the de-padded schedule exposed a LATENT race in `InstanceActorChildAttributeRaceTests` (~1-in-6 under load, A/B'd 14/14 green on the unmodified tree so pre-existing) — fixed with a bounded-deadline resolution retry; pattern recorded as row 77. | Found by the R5 sweep (`9fb52153`) | Closed 2026-08-15. | — |
| 41 | **LocalDb (scadaproj) sync-session tests flake under parallel load** (scadaproj repo, not this one) | Found by R1; confirmed pre-existing at base `f627cee` (1 failure in 3 full runs) | `SyncSessionTests.Handshake_DigestMismatch_FailsClosed` and `BatchBudgetTests.Pump_RowLargerThanTheBudget_IsSentAloneRatherThanStallingTheStream` intermittently time out. Root cause looks structural, not a timing pin: on a handshake fault, `RunAsync`'s catch cancels the writer loop, which can kill the session's OWN Handshake before it transmits — the healthy peer then waits out its full 15 s cap for a message that never comes. Product-adjacent (the same early-cancel could delay a real peer's fail-closed verdict), so worth a look beyond the tests. | Next LocalDb session |
| 42 | **`build/push.sh`-class publish bug across the family repos** | Found by R1 the hard way (incident: 25 unrelated packages briefly published to the feed, deleted + verified same session; script fixed in scadaproj `0038d46`) | The bug shape: `dotnet pack -c Release -o ./artifacts` with no project argument and no `cd`, so the script packs the CALLER'S CWD. Fixed in LocalDb's script (cd to own repo root, name the solution, absolute output path). Peer repos' publish scripts (MxAccessGateway, HistorianGateway, OtOpcUa, others in scadaproj) should be checked for the same shape — the recovery only worked because the token happened to carry delete rights. | Before anyone runs a family publish script from outside its repo, or next umbrella-repo session |
| 42 | **`build/push.sh`-class publish bug across the family repos** | Found by R1 the hard way (incident: 25 unrelated packages briefly published to the feed, deleted + verified same session; script fixed in scadaproj `0038d46`) | The bug shape: `dotnet pack -c Release -o ./artifacts` with no project argument and no `cd`, so the script packs the CALLER'S CWD. Fixed in LocalDb's script (cd to own repo root, name the solution, absolute output path). **AUDITED 2026-08-15 (read-only, this wave): SEVEN scadaproj scripts still carry the exact shape** — `ZB.MOM.WW.Secrets/build/push.sh`, `ZB.MOM.WW.Theme/build/{pack,push}.sh`, `ZB.MOM.WW.Audit/build/{pack,push}.sh`, `ZB.MOM.WW.Auth/build/{pack,push}.sh` (all `dotnet pack -c Release -o ./artifacts` with no project argument and no `cd`, plus a CWD-relative `nuget push "./artifacts/*.nupkg"` that would publish whatever sits in the caller's artifacts dir). MxAccessGateway's `scripts/pack-clients.ps1` is SAFE (`$PSScriptRoot`-anchored, explicit csproj paths); HistorianGateway and OtOpcUa have no pack scripts. Fix = copy LocalDb's `0038d46` pattern into the seven scripts; lives in the scadaproj repo, outside this wave's edit scope. | Before anyone runs a family publish script from outside its repo, or next umbrella-repo session — **now with the exact seven-script fix list ready** |
| 43 | ~~**Central UI import wizard leaks a `BundleSession` slot on every abandoned import — imports lock out for 30 min**~~ **RESOLVED 2026-08-15 (`70838982`).** New `IBundleImporter.Abandon(sessionId)` does the same zero-then-remove release `ApplyAsync` already did (T-007 discipline), idempotent (double/unknown/expired ids are no-ops) and only ever given the caller's own random-GUID id, so it cannot evict a concurrent circuit's session. Wired into EVERY abandonment path: wizard new-upload, `TryLoadAsync` pre-overwrite, Back-to-Upload/`ResetSessionState`, 3-strike lockout reset, `Dispose` — plus two CLI leaks the original row didn't know about: `transport preview` released nothing at all (every preview burned a slot for 30 min) and `transport import` stranded the session on pre-apply aborts; both now release in a `finally`. `BundleSessionEvictionService` is documented as the TTL backstop, not an abandonment path. Fail-before-fix stash-verified (3 of 4 new bUnit tests fail pre-fix). 10 new tests. | Found live by R6 | Closed 2026-08-15. | — |
| 44 | ~~**`BundleUnlockRateLimiter` is dead code — the documented per-IP unlock throttle is not enforced**~~ **RESOLVED 2026-08-15 (`5ab26a42`).** `BundleImporter.LoadAsync` now registers every unlock attempt against `MaxUnlockAttemptsPerIpPerHour` (10). Ordering is load-bearing: AFTER the per-bundle 3-strike check (hammering a locked bundle spends no hourly budget) and BEFORE the decrypt (a throttle denial burns no per-bundle strike, so the throttle can't be abused to lock out a legitimate import); successful unlocks count too. Denial = `BundleUnlockThrottledException` (placed in Commons so the management surface returns a curated message). Client key: Central UI via a scoped `CircuitClientAddressProvider` snapshotting the IP at circuit establishment; CLI via additive `ManagementEnvelope.ClientIp`. No attributable IP falls into one shared `<unattributed>` bucket — stricter, never a bypass; per-client isolation is only real behind a ForwardedHeaders-trusted proxy (same caveat as the login throttle, documented). Fail-before-fix verified (4 of 6 new tests fail with the limiter neutralised; 2 negative controls). | Found by R6 | Closed 2026-08-15. | — |
| 45 | ~~**Notification Outbox dispatcher shares one `DbContext` across up to 4 parallel deliveries**~~ **RESOLVED 2026-08-15 (`9e769735`).** `_adaptersCache`/`_adaptersScope`/`ResolveAdapters()` deleted; the typed adapter is resolved inside the per-delivery DI scope `DeliverGatedAsync` already opens (static `ResolveAdapter` preserving DI last-wins + the null-adapter park path), so each concurrent delivery gets its own adapter and `DbContext`. Nothing shared was lost: the Email adapter builds a fresh SMTP client per attempt from an injected factory, the SMS adapter draws from the singleton `IHttpClientFactory`, and CentralUI's adapter-gated Type selector (`NotificationChannelCatalog`) is an independent scoped consumer, untouched. Fail-before-fix stash-verified (simulated DbContext throwing EF's exact second-operation exception under a rendezvous-guaranteed overlap: 1 of 2 rows persisted pre-fix; 1-vs-4 distinct adapter instances). Root cause of note: the bug was INTRODUCED by code-review finding NotificationOutbox-006's caching "fix" (2026-05-28), whose own regression test pinned the broken behavior and predates bounded parallel delivery — that finding is marked SUPERSEDED. NotificationOutbox 153, AuditLog 383 green. | Found by R6; observed live on rig central-b | Closed 2026-08-15. | — |
| 46 | **CLI `audit-config query` returns `{}` for every filter combination** *(Low)* | Found by R6 against the rig; underlying `AuditLogEntries` rows verified present | Possibly a CLI output-shape/serialization issue; not investigated (out of R6 scope). | Next CLI session |
| 46 | ~~**CLI `audit-config query` returns `{}` for every filter combination**~~ **RESOLVED 2026-08-15 (`09108018`).** Not a CLI bug: `ManagementActor.HandleQueryAuditLog` returned the repository's ValueTuple `(Entries, TotalCount)` straight to `SerializeResult`; System.Text.Json writes properties only and a ValueTuple's `Item1`/`Item2` are **fields**, so the whole payload serialized to the literal `{}` — every filter combination looked empty with no error anywhere in the stack. Fixed with a named Commons record `AuditLogQueryResult` (mirroring `SecuredWriteListResult`) plus a class-wide guard: `SerializeResult` now throws on any `ITuple` result, so the next tuple-returning handler fails loudly instead of silently emitting `{}`. Fail-before verified (test asserted exactly `"{}"` pre-fix); live re-verified on the rig (`totalCount 2725` unfiltered, `712` for `--entity-type Template --action Delete`). The only other tuple-returning repo method in Commons (`INotificationOutboxRepository.QueryAsync`) is never serialized. | Found by R6 against the rig; underlying `AuditLogEntries` rows verified present | Closed 2026-08-15. | — |
| 47 | ~~**`SiteStreamGrpcServer` duplicate-stream replacement disposed another handler's `CancellationTokenSource`**~~ **RESOLVED 2026-08-15 (found and fixed in-flight by R2, `e6842c10`).** `SubscribeInstance`'s duplicate-prevention path called `Cancel()` AND `Dispose()` on the replaced stream's CTS, which belongs to that still-running handler's `using var` and is still read — escaping as an unhandled `ObjectDisposedException` under scheduling pressure (confirmed pre-existing on `2b74851f`; load-only, passed 10/10 isolated). Fixed to cancel-only (the owner's `using` disposes exactly once), `Cancel` wrapped for the converse race, mirroring `CancelAllStreams()`. Deterministic regression test via a gate inside the setup window, fail-before/pass-after verified by reinstating the `Dispose()` as a negative control. Recorded here (though closed) because it is a product-bug instance of the disposal-race class, useful precedent for future stream-lifecycle work. | Closed 2026-08-15. |
| 48 | **Docker rig has no running site instances — live stream probes have no traffic** | Found during the R2 close-out live probe (2026-08-15) | All 7 seeded instances are `soak-*` rows whose deploy fails pre-deployment validation (34 errors: ScriptCompilation ×4, ConnectionBinding ×30 — their bound data connections no longer exist on the sites), and `debug snapshot` returns `instanceNotFound`; `scadabridge_site_connection_up` is 0 on all six site nodes. Consequence: the R2 batching histogram (`scadabridge.site.stream.batch_size`) could not be observed live — batching verification rests on its integration/wire-skew/latency tests, which are thorough. Fix: re-seed a small working instance + data connection set (OPC UA sim is still up) so future live probes have traffic. | Next rig session, or before any live probe that needs event traffic |
| 49 | **Sweep code-review "Resolved" entries that cached or lifetime-widened a DI-scoped service** *(Low, process)* | Found while fixing row 45: NotificationOutbox-006, a Low perf nit, introduced the row-45 Medium correctness bug, and its regression test actively pinned the broken behavior — written before `MaxParallelDeliveries` existed and never revisited when parallel delivery landed | The failure shape generalizes: a review resolution that widens a scoped service's lifetime is correct at the time and silently invalidated by a later concurrency feature. Sweep `code-reviews/*/findings.md` resolved entries for caching/lifetime-widening of scoped services and re-check each against features added after its date. | Next code-review or test-infra session |
| 50 | **No guard against capturing a scoped service on a singleton actor (captive dependency)** *(Low)* | Found while fixing row 45 | The actors take root `IServiceProvider` rather than typed services, which defeats .NET's built-in scope validation — the row-45 class of bug is invisible at build time. Cheap mitigations: `ValidateScopes`/`ValidateOnBuild` on the central host's provider in Development, and/or a test asserting no `ReceiveActor` field holds an `IServiceScope`. | Next DI/host-hardening session |
| 51 | **`BundleLockedException` reaches the CLI as "An internal error occurred"** *(Low, pre-existing)* | Found while fixing row 44 | `ManagementService` doesn't reference `Transport`, so `MapFault` can't curate the 3-strike lockout — a CLI operator gets a correlation id, not the reason. Row 44 solved this shape for the throttle exception by placing it in Commons; `BundleLockedException` needs the same move (public type relocation, low risk, deliberately kept out of the row-44 diff). | Next Transport/CLI session |
| 52 | **The unlock throttle's counters are per-node and per-process** *(Low, by design but recorded)* | Row 44 implementation note; documented in `docs/components/Transport.md` | Each central node has its own limiter, so the effective fleet cap is 10/hour/IP per node, and a restart clears the window. Acceptable for the threat model today; a shared store is the hardening if it ever matters. | A hardening pass that wants a fleet-wide cap |
| 53 | **`MaxConcurrentImportSessions` has no observability** *(Low)* | Found while fixing row 43 — R6 could only infer the leak from rotating test failures | Nothing counts open sessions or logs when `Open` rejects at the cap. A gauge (mind the `ObservedMeters` allowlist) or a Warning log on rejection would make a recurrence diagnosable in seconds. | Next Transport or observability session |
| 48 | ~~**Docker rig has no running site instances — live stream probes have no traffic**~~ **RESOLVED 2026-08-15 (`b5e6821c`).** Working set re-seeded on site-a entirely via the CLI (`soak-*` rows left alone): template `RigProbe` + instance `rig-probe-1`, bound to the OPC PLC sim's built-in moving tags (`SpikeData`, `RandomSignedInt32`) plus a 5s heartbeat script. Deploy Success; `debug snapshot` returns live Good-quality data; `health site` shows `OPC PLC Simulator: Connected`, `good:2 bad:0 uncertain:0`; a 60s `debug stream` produced 239 events; and **`scadabridge.site.stream.batch_size` recorded live for the first time** (count 70 / sum 238, batches to ~10 events) — the R2 batching histogram is now observable on the rig, which was this row's purpose. Reproducible via `docker/seed-live-probe.sh`. Two diagnostic corrections worth keeping: the custom tags in `infra/opcua/nodes.json` are declared but **not simulated** by opc-plc (a healthy-looking connection with zero traffic — bind to the built-ins: `SpikeData`, `RandomSignedInt32`, `StepUp`, `FastUInt1..5`, `SlowUInt1..5`); and `scadabridge_site_connection_up` counts open central→site gRPC subscriber streams, not data connections, so this row's original "0 on all six site nodes" was healthy steady state, not the fault (row 54). Close-out postscript: the seeding agent's live verification ran against worktree-homed data dirs (row 68), so after the orchestrator's redeploy from main the config was absent at boot and re-fetched by the startup reconcile — which exposed row 67 — and `rig-probe-1` was re-verified live on the main checkout's data dirs (snapshot Good ×3, heartbeat ticking, sim values moving). | Found during the R2 close-out live probe (2026-08-15) | Closed 2026-08-15. | — |
| 49 | ~~**Sweep code-review "Resolved" entries that cached or lifetime-widened a DI-scoped service**~~ **RESOLVED 2026-08-15 — swept, no further bug found (`75520bee`).** All 576 resolved findings across the 25 component findings files were re-read for the row-45 generalization (a resolution that caches, hoists, or widens the lifetime of a DI-resolved service, later invalidated by a concurrency/lifetime feature). Nine matched or nearly matched the shape; **NotificationOutbox-006 is the only one that ever became a bug** and was already fixed as row 45. The other eight verify STILL-SAFE or SUPERSEDED-BENIGN — the two with real concurrency nearby each anticipated it explicitly (`DeploymentService`'s `dbGate` serialises the one repository touch inside the bounded-parallel Send phase; `ArtifactDeploymentService` builds all per-site commands serially before fanning out, commenting "DbContext is not thread-safe"). An independent product-code cross-check confirms the pattern is absent: no `IServiceScope` is stored on a field anywhere in `src/` (every long-lived holder takes `IServiceScopeFactory` and opens a scope per operation), every `Task.WhenAll`/`Parallel.*` site either touches no scoped service or opens a per-operation scope, and no scoped-registered service is captured by a singleton. One stale archive corrected: NotificationOutbox-004 was marked Resolved with an `_Unresolved._` body; the real resolution is now recorded. Near-miss shapes recorded as rows 58–60. | Found while fixing row 45 | Closed 2026-08-15. | — |
| 50 | ~~**No guard against capturing a scoped service on a singleton actor (captive dependency)**~~ **RESOLVED 2026-08-15 (`00a804e6`).** Both mitigations shipped, and they are complementary. (1) `ValidateScopes`/`ValidateOnBuild` in Development on **both** role branches via `Host/ServiceProviderValidation.cs` — written out rather than left to `CreateBuilder`'s silent default; production stays unvalidated deliberately (fail-closed at boot). Switching it on surfaced **two real Site-role defects**: `ReconcileService` and `AuditLogKpiSampleSource` were registered on sites but need central-only repositories, so neither could ever be constructed there — one unconstructible descriptor fails `ValidateOnBuild` for the entire site graph, which is exactly why `SiteHealthEndpointTests` had quietly pinned `UseEnvironment("Production")` to dodge validation. Both moved to central-only registrations (`AddCentralCommunication()` / `AddAuditLogCentralKpi()`); the workaround is gone and the site graph now boots validated. (2) `CaptiveDependencyGuard` reflects over all 28 shipped actor classes and fails on any instance field (backing fields included) typed `IServiceScope`, `DbContext`, or Scoped-in-the-real-containers — the scoped set is derived from the built Central and Site graphs, so a lifetime change auto-re-scopes the check; `IServiceProvider`/`IServiceScopeFactory` fields are allowed (the correct pattern). Proven non-vacuous via permanent positive-control dummy actors. No live violation; the two test-only `_injectedRepository` ctor fields allowlisted with written safety arguments plus a staleness self-test (row 61). Doc: REQ-HOST-11 in `Component-Host.md`. | Found while fixing row 45 | Closed 2026-08-15. | — |
| 51 | ~~**`BundleLockedException` reaches the CLI as "An internal error occurred"**~~ **RESOLVED 2026-08-15 (`b449fe20`).** Relocated to `Commons/Types/Transport/` beside `BundleUnlockThrottledException` (row 44's pattern); both consumers already carried the namespace, so no call-site edits. `ManagementActor.HandlePreviewBundle`/`HandleImportBundle` now rethrow it as `ManagementCommandException`, so CLI and UI print the lockout reason instead of a correlation id. Fault-mapping tests mirrored from row 44 on both handlers. Follow-up in the same diff (`73f518d6`): deleted the dead `BundleUnlockRateLimitedException` (superseded by row 44, zero references — three similarly-named unlock exceptions with one inert was a live trap). | Found while fixing row 44 | Closed 2026-08-15. | — |
| 52 | ~~**The unlock throttle's counters are per-node and per-process**~~ **RESOLVED 2026-08-15 (`677263af`).** The trailing-hour window now lives in the central MS SQL `BundleUnlockAttempt` table — the HA store both central nodes already share, so no new infra, port or consensus. `MaxUnlockAttemptsPerIpPerHour` is a true fleet cap and survives a node restart; config keys, semantics and the rig's 500 override are unchanged, as are all row-44 behaviours (lockout checked first so locked bundles spend no budget; denials burn no strike and are not recorded; `<unattributed>` bucket; attempts-not-failures). Three recorded trades: **fails open** (store error → Warning naming client + cap, attempt permitted — availability of legitimate imports outranks a best-effort cap over an already-authenticated Administrator-gated surface; a process-local fallback was considered and rejected as it would still block an operator's legitimate 11th import); **approximate** (count-then-insert is not atomic across nodes; overshoot bounded by in-flight count, self-corrects within the window — row 64); **opportunistic prune on the unlock path** (CAS-claimed ≤ every 15 min per process, ≤1000 rows, retains 2× the window so a prune can't race a count — row 65) rather than a new background service. EF migration `20260815154803_AddBundleUnlockAttemptTable` + hand-written production script `docs/plans/sql/AddBundleUnlockAttemptTable.sql`. Integration test proves two limiter instances over one shared store enforce one combined window, restart durability, and store-failure fail-open. | Row 44 implementation note; documented in `docs/components/Transport.md` | Closed 2026-08-15. | — |
| 53 | ~~**`MaxConcurrentImportSessions` has no observability**~~ **RESOLVED 2026-08-15 (`372ea4df`).** Gauge `scadabridge.central.transport.import_sessions.open` on the `ZB.MOM.WW.ScadaBridge` meter — already in `SiteServiceRegistration.ObservedMeters` (the single shared allowlist source for both roles; no appsettings pins one), so no allowlist change was needed and the series is not silently dropped. Provider-backed like the S&F queue-depth gauge: registered by `BundleSessionEvictionService.StartAsync`, identity-checked-cleared on `StopAsync`. `IBundleSessionStore.OpenSessionCount` counts only **unexpired** sessions, so the gauge doesn't read "pinned at the cap" for a sweep interval after a burst drains. `BundleSessionStore.Open` logs Warning with open count, cap and TTL on a genuine cap hit only (a store full of expired sessions is reclaimed silently). No alertable rejection counter yet — row 66. | Found while fixing row 43 — R6 could only infer the leak from rotating test failures | Closed 2026-08-15. | — |
| 54 | ~~**`scadabridge_site_connection_up` is misleadingly named** *(Low, observability)*~~ **RESOLVED 2026-08-15 (`b5f120c3`).** Renamed to `scadabridge.site.stream.subscribers.active` (a test pins the new name AND asserts the old one is gone; doc/script references updated) and BOTH real data-connection gauges added: `scadabridge.site.data_connections.connected` / `.total` — the denominator ships with the numerator because neither alone separates "device down" from "nothing configured"; `connected < total` is the alert condition. Sourced from the DCL's four state transitions, deliberately NOT from the health report, so the metric survives a dead report pipeline (and stays clear of the concurrent row-55 fix in `SiteHealthCollector`). All on the already-allowlisted `ZB.MOM.WW.ScadaBridge` meter. Expected rig shape: connected==total on the active site node, 0/0 on the standby (healthy — connections open on the active node only). | Row-48 close-out (2026-08-15) — the row's original mis-diagnosis traced directly to this | Closed 2026-08-15. | — |
| 55 | ~~**`tagResolutionCounts` reads `{totalSubscribed:0, successfullyResolved:0}` for a connection with 2 live Good-quality subscribed tags** *(Low)*~~ **RESOLVED 2026-08-15 (`5da27e05`).** All three suspected reporting-path hypotheses were WRONG — live debugging showed the zeros and the `good:2` on the SAME report from the SAME collector on the active node, ruling out standby shadowing, keying races, and the central merge in one stroke. The real bug is a DCL omission, not a regression: `DataConnectionActor.HandleSubscribeCompleted` — the first-time, everything-resolves-cleanly subscribe path — was the missing FOURTH `UpdateTagResolution` push site. `BecomeConnected`'s legitimate 0/0 push fires before the stashed subscribe unstashes (proven in live logs: "Entering Connected state" follows the subscribe line), `ScheduleTagResolutionRetry` cancels outright at zero unresolved, and any reconnect masks the bug via the batch path — so the diagnostic was blindest on a perfectly HEALTHY connection, exactly when it should read clean rather than empty. One mirrored push + regression test (stash-verified fail-before: exactly one `0/0` call received pre-fix while the actor's own `GetHealthReport` already said 2/2, pinning the actor-correct/collector-stale asymmetry). Also corrected `Component-DataConnectionLayer.md`, which claimed the counts were "derived at report time" — they are pushed, and a cached push is only as fresh as its last push site. | Row-48 live probe; root-caused live 2026-08-15 | Closed 2026-08-15. | — |
| 56 | ~~**The rig's central nodes expose no Prometheus endpoint** *(Low, rig config)*~~ **RESOLVED 2026-08-15 (`103f2f19`).** Premise refined on inspection: central already served `/metrics` on its web port (anonymous, no fallback policy) — what was missing was the dedicated 8084 listener the documented docker-exec scrape recipe targets. Both rigs' central appsettings now set `MetricsPort`, and the Host binds it as a PRESENCE check, not a defaulted read — `NodeOptions` defaults to 8084, so a defaulted read would make every existing production central surprise-bind a new port on upgrade, and port-in-use is a hard boot failure. | Row-48 live probe | Closed 2026-08-15. | — |
| 57 | ~~**`audit-config query` has no CLI table projection** *(Low, cosmetic)*~~ **RESOLVED 2026-08-15 (`b2cab59c`).** `AuditConfigTableProjection` mirrors `TemplateTableProjection`: unwraps the `{entries, totalCount}` envelope to rows (`id, timestamp, user, action, entityType, entityName, entityId`), drops the fat `afterStateJson`, adds `--detail` to skip projection (matching `template list`), and surfaces `totalCount` as a trailing note row (`showing N of M — use --page/--page-size`) emitted ONLY when the page is a strict subset, with no `id` cell so it can't be mistaken for an entry. The deprecated `audit-log` alias shares the command tree and gets the projector too. Tests serialize the REAL `AuditLogQueryResult`/`AuditLogEntry` types, so a server-side property rename fails the test instead of silently blanking a column. Stale-README follow-up recorded as row 76. | Row-46 fix (2026-08-15) | Closed 2026-08-15. | — |
| 58 | ~~**`IDeploymentStatusNotifier` singleton retains delegates targeting circuit-scoped Blazor components** *(Low)*~~ **RESOLVED 2026-08-17 (`2807b5a3`).** Still exactly one subscriber (`Deployments.razor`), but the cited pin was BLIND: `Deployments_Dispose_UnsubscribesFromNotifier` was satisfied by the `_disposed` guard alone — mutation-verified (with the `-=` deleted, the old test stayed green while the leak was wide open). Closed by two real pins: `DeploymentsPushUpdateTests.Deployments_Dispose_ActuallyDetachesTheDelegate_NotJustGuardsIt` (structural — the invocation list must shrink to 0) and `EveryCentralUiSubscriberToTheNotifier_UnsubscribesAndIsDisposable` (a source scan pinning this row's own trigger: any new subscriber without a matching Dispose unsubscribe fails the suite). Both fail on the mutation. | Row-49 sweep (CentralUI-006 re-verification) | Closed 2026-08-17. | — |
| 59 | **`SiteCallAuditActor` per-tick scope spans all sites × ≤50 upsert pages** *(Low)* | Row-49 sweep (SiteCallAudit-009 re-verification); re-verified 2026-08-17 (Low-tail sweep — loop still serial, line numbers drifted to `SiteCallAuditActor.cs:528-545`) | Safe only because site reconciliation is a serial `foreach … await`; one scope across a parallel fan-out would be row 45 again. 2026-08-17: the missing in-code constraint remark was added at the loop (`2807b5a3` — "serial by design … do not fan out"); a source-scan pin was judged too brittle, so the comment + this row remain the tripwire. | `ReconcileSiteAsync` is ever fanned out with `Task.WhenAll` |
| 60 | ~~**`ArtifactDeploymentService.GlobalArtifactSnapshot` holds EF-materialised entities across the parallel per-site send** *(Low)*~~ **RESOLVED 2026-08-17 (`2807b5a3`) — with a premise correction so the row is never reopened on the wrong reasoning.** The row's premise was factually wrong: `GlobalArtifactSnapshot` holds pure Commons DTOs, not EF entities (`ArtifactDeploymentService.cs:170-212`); the EF state that actually crosses the fan-out is the tracked `Site` entities the per-site lambda closes over. The safety conclusion still holds — commands are built serially before the fan-out and lazy loading is confirmed absent — and the load-bearing half of the trigger is now permanently pinned by `ConfigurationDatabase.Tests.AddConfigurationDatabase_DoesNotEnableLazyLoading`. | Row-49 sweep (DeploymentManager-023 re-verification) | Closed 2026-08-17. | — |
| 61 | ~~**`_injectedRepository` test-mode constructors are one call site from re-introducing row 45** *(Low)*~~ **RESOLVED 2026-08-15 (`92fe5acc`).** Preferred fix done: both actors now have exactly ONE constructor (`IServiceProvider`); 12 test call sites folded onto stub providers via two `TestSupport` helpers. Also deleted as newly-dead: `OnCachedTelemetryWithoutDualWriteAsync`, `SiteCallAuditActor`'s `_backgroundTimersEnabled` (a constant once the test ctors went), and the private `SerializedRepository` — moved into the test project, since production resolves a fresh repository per scope and the concurrency hazard existed only for fixtures sharing one instance. `CaptiveDependencyGuard` allowlist is now EMPTY; the staleness self-test replaced by two stronger pins — `Allowlist_IsEmpty` plus `FormerlyAllowlistedActors_NowScanClean`, which scans both actors with the allowlist DISABLED, so emptiness is evidence the fields are gone, not that the scan stopped looking. Control: re-adding a field fails both pins and the real scan. | Row-50 guard allowlist | Closed 2026-08-15. | — |
| 62 | ~~**Five SiteRuntime actors resolve `ISiteEventLogger` off the root provider without a scope** *(Low)*~~ **RESOLVED 2026-08-16 (`9d668861`) — pinned, and the row undercounted: there are SIX sites.** `SiteEventLoggerLifetimePinTests` (Host tests, registration-only inspection, no provider built) pins TWO facts — the interface registration is a *factory* forwarding to concrete `SiteEventLogger`, so pinning the interface lifetime alone doesn't prove callers share an instance; the concrete registration is pinned too, and flipping only the concrete to Scoped is caught ONLY by the second pin (verified both directions, registration restored). The pin's failure message names all six root-resolve sites: the five in this row plus `SiteRuntime/Scripts/ScriptRunLauncher.cs:224` — the run body WP3.1 lifted verbatim out of the deleted execution actors, which inherited the pattern unnoticed. | Row-50 guard allowlist | Closed 2026-08-16. | — |
| 63 | ~~**Rig site containers run without `ASPNETCORE_ENVIRONMENT`, so the row-50 Development validation never fires there** *(Low, rig config)*~~ **RESOLVED 2026-08-15 (`0d2d6de6`).** Set on all six `docker/` site containers and both `docker-env2` site containers, mirroring the centrals; both compose files re-parsed to prove well-formedness; the now-stale `Program.cs` comment corrected. A captive dependency in the site graph now fails the container at boot on the rig, not just in CI. | Row-50 implementation | Closed 2026-08-15. | — |
| 64 | **The fleet-wide unlock throttle's count-then-insert is not atomic across central nodes** *(Low, accepted by design)* | Row-52 implementation trade, documented in both Transport docs (re-verified 2026-08-17: `Component-Transport.md:341`, `docs/components/Transport.md:140`, plus the code XML doc) | Simultaneous attempts can overshoot `MaxUnlockAttemptsPerIpPerHour` by the number in flight; bounded and self-correcting within the window. A conditional single-statement insert would close it at the cost of provider-specific raw SQL. | A threat model needing a hard cap rather than a rate damper, or evidence of a real brute-force exploiting the overshoot |
| 65 | ~~**`BundleUnlockAttempt` rows are pruned only by nodes actively serving unlocks** *(Low)*~~ **RESOLVED 2026-08-17 (`2807b5a3`) — the revisit trigger had FIRED.** Transport gained exactly the central scheduler the trigger named: `BundleSessionEvictionService` (central-only `BackgroundService`; `AddTransport` is called only from the central branch of `Program.cs`). The prune tick now rides it, so an idle cluster's rows age out without an unlock ever occurring; the opportunistic unlock-path prune remains. Operational note: the service now has two maintenance duties, deliberately in separate `try` blocks so neither can mask the other. Tests: `FleetWideUnlockThrottleTests` additions + `BundleSessionObservabilityTests` extension, both green. | Row-52 implementation trade | Closed 2026-08-17. | — |
| 66 | ~~**No counter for import-session cap rejections** *(Low)*~~ **RESOLVED 2026-08-15 (`90ce0999`).** `scadabridge.central.transport.import_sessions.rejected` on the existing allowlisted meter, emitted ONLY at the post-eviction recheck — the cap is tested twice and a store full of expired sessions reclaims and succeeds, so counting the first test would fire routinely on any node whose sweep hadn't run; duplicate-session-id re-`Open` is also excluded. New tests use delta-over-scoped-action with unique per-test keys (the pre-existing gauge tests' absolute asserts are row 74). | Row-53 implementation note | Closed 2026-08-15. | — |
| 67 | ~~**Startup reconcile stores a fetched instance config but does not start the InstanceActor** *(Medium-Low, availability)*~~ **RESOLVED 2026-08-15 (`482165cb`).** Root cause: InstanceActors are created from exactly two paths — the boot-time SQLite load in `DeploymentManagerActor.PreStart` and an explicit deploy/enable command — and the reconcile pass fires ~5s AFTER the boot load with a pure file write (`StoreDeployedConfigIfNewerAsync`) as its only side effect. Fix: reconcile collects the enabled instances it stored and Tells a new `EnsureInstanceStartedCommand` to the DM **singleton proxy**, so the actor is created under the singleton no matter which node's pass found the gap. Handler is advisory and idempotent: no-op if already running; re-reads the durable row (authority on config body AND `is_enabled`, so a disable landing after the fetch wins); defers to `_deployWarms`/`_terminatingActorsByName`/`_inFlightDeployments` so deploy-gate ordering and all-or-nothing-per-instance are untouched (it starts an already-persisted config, never applies one); quiet no-op on an absent row (normal on an unreplicated site whose peer healed its own file). The heal is no longer silent — a Warning site event `Instance <X> started by startup reconcile self-heal` fires. Fail-before by behavioral revert: 3 of 8 new tests fail pre-fix, including the exact live symptom (`debug snapshot` → `instanceNotFound`); the 5 negative guards (disabled / failed-fetch / unknown / idempotence) green in both states. SiteRuntime 613/613, Host 504/504. Sibling gaps found during the fix recorded as rows 72/73. | Proven live during the rows-46–53 close-out (2026-08-15) | Closed 2026-08-15. | — |
| 69 | ~~**Both site-a nodes' LocalDb files corrupt — replication down, one node crash-looping**~~ **RESOLVED 2026-08-15 (operational recovery, no code change).** Found live during this wave: site-a-b threw `SQLite Error 11: 'database disk image is malformed'` on every sync call (replication `connected: false`, backlog 892, faulted session retried ~every 60s) and, after the redeploy, site-a-a crash-looped (SIGABRT ×12) on the same error at `SqliteLocalDb` construction — the new boot-time re-registration reads pages the old build never touched, so site-a-a's corruption (present but latent) became fatal only on upgrade. Recovery: pair stopped together, both files quarantined (`*.corrupt-2026-08-15`, kept for forensics; 394 MB / 416 MB), fresh files reseeded — site-a-b via its startup reconcile (which live-proved the row-67 fix, see row 67) and CDC snapshot resync; converged to `connected: true`, backlog 0, with the row-27 schema change proving the digest handshake green. Suspected cause is host-level, not product: see row 79. | Found live by the row-55 debug + this wave's close-out redeploy | Closed 2026-08-15. | — |
| 70 | **`auto-down-unreachable-after` is deprecated — Akka warns "keep-majority will be used instead" on every boot** *(Medium info — verified NOT currently true, but a live removal risk)* | Row-55 live session; investigated + drill-verified at this wave's close-out | The warning is real (fires at Cluster start on every node, Akka.NET 1.5.62 — same version as the 2026-07-21 decision, so nothing changed underneath; it fires on the presence of the `auto-down-unreachable-after` key). **Drill-proven that the substitution does NOT happen when `downing-provider-class = "Akka.Cluster.AutoDowning"` is EXPLICIT (as `BuildHocon` sets):** `DRILL_MODE=active` kills central-a (oldest AND lowest address) — under keep-majority a 2-node even split keeps the lowest-address side, so surviving central-b would have downed ITSELF (total outage); instead central-b observed OldestChanged → became Oldest, took over all singletons, and the victim rejoined ready 2s after restart (2026-08-15 drill, post-merge rig). The 2026-07-21 decision therefore still holds exactly as documented. The residual risk is the deprecation itself: a future Akka.NET bump can delete `AutoDowning`, silently or loudly breaking the availability-first strategy. | **Any Akka.NET version bump** — re-run `DRILL_MODE=active bash docker/failover-drill.sh` and re-read the release notes before accepting it |
| 71 | ~~**Standby nodes log the active-node health check at ERR every 10s** *(Low, log noise)*~~ **RESOLVED 2026-08-16 (`8b761841`).** One narrowly-keyed Serilog `Filter.ByExcluding` in `LoggerConfigurationFactory`: `Level == Error` AND the `DefaultHealthCheckService` category AND `HealthCheckName == "active-node"` — check-NAME keying is what preserves the constraint (`localdb`/`akka-cluster`/`database`/`required-singletons` keep their Error lines, and active-node has no error path of its own on the active side, so the only suppressible event is a standby reporting it's a standby). Rejected on evidence: `MinimumLevel.Override` can only raise a floor (can't rewrite Error→Information), and reporting `Degraded` would change the HTTP status `/health/active` publishes, which is load-bearing for Traefik. Bonus reach: the CENTRAL standby was logging the same Error 2x as often (Traefik probes `/health/active` every 5 s) — covered by the same filter. Fail-before: filter disabled → `StandbyActiveNodeVerdict_IsNotLogged` fails; the 9 must-still-log constraint tests pass either way, proving narrowness. Side-finding: the observed ~10 s SITE-side cadence comes from an out-of-repo prober (nothing in this repo probes a site's `/health/active`), and CLAUDE.md's "site nodes map no `/health/*`" claim was stale — corrected. | Row-55 live session (2026-08-15) | Closed 2026-08-16. | — |
| 72 | ~~**Reconcile refreshes a STALE config but does not restart the running InstanceActor** *(Medium-Low — same silent-divergence class as row 67, arguably wider blast radius)*~~ **RESOLVED 2026-08-16 (`e606b24a`, with row 78 — one fix, the DM instance sweep).** Design decision made and recorded (`docs/plans/2026-08-15-instance-sweep-self-heal-design.md`): **converge, don't merely warn** — the sweep re-applies the stored config through **`HandleDeploy`**, the exact path a central redeploy uses, so the compile gate, warm-then-gate ordering, redeploy-supersede, static-override/native-alarm reset and all-or-nothing-per-instance are the production ones (a stop still does not cancel an in-flight run). Rationale: the stored row IS what central successfully deployed; surface-only leaves the operator to hand-issue the very redeploy the sweep would perform. Ordering hazards checked: the sweep never acts on an instance a deploy pipeline owns; a gate-rejected restart is inert, not partial; the synthetic deploy replies to `Self`, never `NoSender` (dead letters are a health metric). "Nothing surfaces it" is now false in BOTH paths — even a failed restart yields two `deployment` site events naming both revisions. Safety property beyond the brief: **one repair attempt per (instance, revision) pair**, so the self-heal can never become a restart loop for an init-crashing actor or a config the gate can never accept. Fail-before: sweep unarmed → `Sweep_RestartsInstance_ServingAStaleRevision` fails showing rev-1's value served while the store holds rev-2. | Row-67 fix session (2026-08-15) | Closed 2026-08-16. | — |
| 73 | ~~**`SiteReconciliationActor` is silently skipped when `IDeploymentConfigFetcher` is null** *(Low)*~~ **RESOLVED 2026-08-16 (`a2f0086f`, with row 80).** New tri-state health-report field **`StartupReconcilerActive`** (`bool?`, matching the `LocalDbReplicationConnected` nullable precedent; additive proto **field 29**, vendored C# regenerated via `docker/regen-proto.sh centralcontrol` with the csproj verified restored): null = not reported (central self-report or pre-row-73 site build), false = booted with no fetcher. Latched once at boot by the Host in both branches. The skip branch was raised **Warning → Error** (no configuration legitimately omits the fetcher on a site node) plus a best-effort `deployment`/`Error` site event so evidence outlives the boot log. Central treatment: a **"No self-heal" Warning badge**, deliberately not a failure — config still arrives via replication while the node is up; what's lost is recovery from a deploy landing while it's down. Only explicit `false` is badged (badging null would light the fleet the moment central upgrades ahead of the sites); offline sites skipped, matching the backlog badge. | Row-67 fix session | Closed 2026-08-16. | — |
| 74 | ~~**`BundleSessionObservabilityTests` gauge tests assert absolute values — latent row-32-class race** *(Low, test-only)*~~ **RESOLVED 2026-08-16 (`98c94588`).** Both gauge facts converted to delta-over-scoped-action via a `ReadGaugeDelta` helper — and two assertions got STRONGER: the post-clear check (`Assert.Equal(0, …)` — satisfied by any unregistered slot) now asserts opens on the detached store move the gauge by nothing (the actual claim), and the compare-and-clear fact gained a negative control (the superseded store's opens must NOT reach the instrument), without which it passed by coincidence whenever both stores held equal counts. Fail-before: a temporary foreign-provider probe returning 7 pre-start — old body fails `Expected: 0 / Actual: 7`, new body passes the identical scenario (probe deleted, not committed). Documented residual: a registrant clobbering the slot BETWEEN the two reads still needs row-32's mutual-exclusion pattern; cannot arise today (sole constructor of `BundleSessionEvictionService` in the assembly). | Row-66 implementation (2026-08-15) | Closed 2026-08-16. | — |
| 75 | ~~**SecuredWrites Approve renders as a red destructive button** *(Low, UI)*~~ **RESOLVED 2026-08-16 (`40944a0d`).** The real defect was deeper than "red means delete": Approve and Reject rendered as IDENTICAL red buttons, so at the moment of a two-person-control decision colour carried no discriminating information — because `danger` was a two-state flag for a three-state concept. Fix: `DialogSeverity` (`Normal→btn-primary`, `Caution→btn-warning`, `Destructive→btn-danger`); **Approve → Caution amber** (executes a real write to live plant equipment — consequential, not destroying), **Reject keeps Destructive red** (terminal for the request; and holding it red keeps the two opposite decisions distinct — dropping it to neutral would just invert the bias). Only `Destructive` inherits row-28's "Delete" label fallback, or row 28 returns through the new door. Implemented as an ADDITIVE overload deliberately: a 5th optional parameter would silently un-match every 4-arg NSubstitute stub (the row-28 gotcha, null Task → NRE); the overload's only breakage was 6 hand-written fakes failing to COMPILE — the loud mode. `DialogState.Danger` became `Severity` (one field can't disagree with itself). Playwright `.btn-danger` selectors stay valid (`Destructive` mapping unchanged). Fail-before: call site reverted to `danger: true` → both Approve pins fail; `Caution→btn-danger` remap → amber styling pin fails. Decision recorded in `Component-CentralUI.md`. | Row-28 audit (2026-08-15) | Closed 2026-08-16. | — |
| 76 | ~~**CLI README's `audit-config query` JSON sample omits `afterStateJson`/`bundleImportId`** *(Low, docs)*~~ **RESOLVED 2026-08-16 (`4ae57bdb`).** Sample now shows both fields with their nullability semantics, and was grown to two entries mirroring the table-output sample below it row-for-row — which is what makes the "the table drops these two fields" sentence checkable. | Row-57 fix (2026-08-15) | Closed 2026-08-16. | — |
| 77 | ~~**Test pattern: reflecting `ActorCell.Actor` right after `ResolveOne` is unordered vs child construction** *(Low, test-infra)*~~ **RESOLVED 2026-08-16 — swept clean, no commit needed.** Full `tests/` sweep: `ActorCell` (1 hit — the already-fixed file's comment), `ActorRefWithCell` (1 — fixed file :135), `.Underlying` (1 — same line), `BindingFlags` near actor/cell (2 — same file), `.Cell` (0), `ResolveOne` (3 files: the fixed one; `SiteAlarmLiveCacheServiceTests:432` resolves-then-`Sys.Stop`s, benign; `ReconcileStartsInstanceTests:363` uses the ref only for `IActorRef` equality, benign), `UnderlyingActor` (6 files — all Akka.TestKit `TestActorRef<T>` via `ActorOfAsTestActorRef`, an ORDERED mechanism, not this pattern). The unordered idiom exists in exactly one place and already carries the bounded-deadline retry. | Row-40 session (2026-08-15) | Closed 2026-08-16. | — |
| 78 | ~~**Configs arriving via LocalDb replication start no InstanceActor — row 67's gap through a side door** *(Medium-Low, availability; proven live 2026-08-15)*~~ **RESOLVED 2026-08-16 (`e606b24a`, with row 72 — the DM instance sweep).** A periodic sweep on the Deployment Manager **singleton** (`ScadaBridge:SiteRuntime:InstanceSweepIntervalSeconds`, default 60, validated `>= 0`, `0` disables — default is live everywhere with no rig config change, which is intended): each tick re-reads the durable rows off-thread and, on the actor thread, starts any **enabled** stored config with no running actor. Arrival-path-agnostic by design — covers replication seeding, post-boot snapshot resyncs, and any future path. A LocalDb change hook was considered and rejected on evidence: it covers only the replication path and only the replicated half of the fleet (site-b/c aren't), AND `ILocalDb` 0.3.0 exposes no change-notification API at all, so a hook means a cross-repo upstream capability to close a site-local gap. Warning site event on a real heal names the sweep as the trigger; silent when there is nothing to do. Fail-before (sweep unarmed): 5 of 10 new tests fail including the exact live symptom ("A replication-seeded config for an enabled instance left no running Instance Actor"); 5 negative guards (disabled / in-flight-deploy deferral / matching-revision no-op / silence / interval-0) green in both states. Design note: `docs/plans/2026-08-15-instance-sweep-self-heal-design.md`. **LIVE-PROVEN at close-out (2026-08-16) in the exact original failure sequence:** site-a pair stopped, node-a's LocalDb file quarantined, pair restarted together — node-a's 08:45:55 reconcile reported "0 fetched" (replication had seeded the fresh store; central rightly saw no gap — the precise side door), and at 08:46:50, one sweep interval later, BOTH enabled instances started with the Warning `Instance rig-probe-1 started by deployment-manager sweep self-heal — its config is present in the deployed-config store but no Instance Actor was running until now`; `debug snapshot` live with no extra restart. The probe also exposed the DCL sibling gap — row 85. | This wave's close-out (2026-08-15) | Closed 2026-08-16. | — |
| 79 | **OrbStack virtiofs is a live corruption + availability hazard for the rig's bind-mounted SQLite files** *(Medium, rig infra — host-level, not product)* — **mitigation chosen 2026-08-16: accept + document (`851b528c`).** The full hazard description + the proven quarantine-and-reseed recovery runbook now live in `docs/deployment/topology-guide.md` (new section), including the `orb restart docker` VM-sleep recovery and the named-volumes/idle-pause alternatives with their trade-offs. Row stays OPEN as the hazard tracker: a recurrence should trigger actually moving to named volumes or disabling idle-pause rather than re-running the runbook a third time. Original evidence: the OrbStack VM idle-SLEPT mid-session (`vmgr … msg=sleep`) leaving docker hung ~20 min until `orb restart docker`, with virtiofs `refresh_nodeid: race with new lookup` ERRORs on wake; both site-a LocalDb files (row 69) corrupted through this bind-mount layer — macOS virtiofs does not give SQLite true fsync durability, and abrupt container kills mid-write compound it. | This wave's close-out (2026-08-15); runbook shipped 2026-08-16 | Runbook shipped; hazard accepted for now | **Next corruption occurrence ⇒ stop accepting: move the rig data dirs to named volumes or disable OrbStack idle-pause** |
| 80 | ~~**Unreplicated sites report NON-NULL `LocalDbReplicationConnected: false` / `LocalDbOplogBacklog: 0` — contradicting the documented nullable semantics** *(Low-Medium, observability correctness)*~~ **RESOLVED 2026-08-16 (`a2f0086f`, with row 73).** Root cause was type-level: `ISiteHealthCollector.SetLocalDbReplicationStatus(bool …)` made null INEXPRESSIBLE for the connected flag, and the Host wired the bridge unconditionally — the replication engine is registered on every site node and idles answering `false`/`0` with no peer. Fix mirrors the `SiteEventLogReplicationCheck` precedent exactly: the Host evaluates `SiteLocalDbSetup.ReplicationIsConfigured` once and passes it as a third provider to `AddLocalDbReplicationHealthBridge` (the SAME captured value now also feeds `SiteEventLogReplicationCheck`, so the two surfaces can't disagree); unconfigured short-circuits to `(null, null)`; the collector widens to `bool?`. The rest of the chain needed NO change — proto already `BoolValue`/`Int64Value` wrappers, `NextBacklogAboveSince` already leaves the anchor unchanged on null, KPI source already conditional: row 33's machinery was correct and had simply never received a null (its null branch was dead code in production until now). `false` now means exactly one thing: configured, peer session down = real fault. Trap that produced the bug, now pinned in tests: configured-ness is `PeerAddress` **OR** `ApiKey` — "no PeerAddress" is NOT "unreplicated" (a passive-half node carries the key alone); the pre-existing Host fixture asserting `false`/`0` was exactly such a node and its assertion is CORRECT — kept with an explaining comment, plus a new genuinely-unreplicated fixture. Fail-before: gate behaviourally reverted → `UnreplicatedSite_HealthReport_ReportsNullForBothLocalDbFields` fails `Expected: null / Actual: False`. | Found at the previous wave's close-out verifying row 33 | Closed 2026-08-16. | — |
| 81 | ~~**The site-event-log storage cap cannot be enforced on a replicated node with an undrained oplog** *(Medium-Low, measured 2026-08-16 during the row-34 work)*~~ **RESOLVED 2026-08-17 (`ddb3c32e`) — but NOT via the row's proposed `dbstat` fix, which is impossible on this stack.** Probed BEFORE implementing: the bundled SQLite 3.53.3 has no `ENABLE_DBSTAT_VTAB` (`pragma_module_list` returns 10 modules, none of them `dbstat`/`sqlite_dbpage`); pinned by `SqliteCapabilityTests`, written to FAIL if a future bump enables it (row 94). Switching SQLite builds rejected — the provider is shared with `ZB.MOM.WW.LocalDb` fleet-wide. The cap instead measures `site_events` + its four indexes via a sampled page-layout model: row width sampled 500 rows from each end per tick (`octet_length`), whole-page table cost (linear models under-estimate near-half-page rows by ~40%), clamped by whole-file logical size (a true upper bound). Index pages count against the cap (~34% of footprint — the 1 GB default was sized against a figure that included them). Measured model error −0.8%/+8.4%/+12.8%, biased high on purpose; cross-checks row 34's independent 393 B/row at 397.2. Guard semantics re-based: (1) unsampleable row width ⇒ refuse to trim (checked BEFORE the cap comparison); (2) the old "size did not decrease" guard now runs on the table-scoped estimate, so replication bookkeeping can no longer trip it — it now means exactly one thing: events arriving as fast as the trim removes them. Bytes-per-row held fixed per tick (re-sampling per batch would jitter the estimate through guard 2 — the same self-defeating measurement as the row itself). LocalDb bookkeeping explicitly NOT managed here (`MaxOplogRows`/tombstone retention own it); topology guide gained a volume-sizing subsection stating the honest budget. Fail-before: reverting to whole-file semantics fails 5 new tests and reproduces the exact original signature (rows destroyed, cap still unmet). SiteEventLogging 104/104, Host 520/520. No config keys added or renamed. Decision memo: `docs/plans/2026-08-17-site-event-log-cap-semantics-decision.md`. Model limitations recorded as rows 92/93. | Row-34 measurement harness; §5 of the measurement doc (postscript appended) | Closed 2026-08-17. | — |
| 82 | ~~**`AuditLog.Tests` flakes under parallel full-solution runs — `MsSqlMigrationFixture` contention** *(Low, test-infra)*~~ **RESOLVED 2026-08-17 (`da23f2dc`) — three mechanisms, and the row's "per-run database name" prescription was already in place (the fixture has minted `ScadaBridgeAuditMigTest_<guid>` per class since it was written).** (1) The headline fingerprint was create+migrate SATURATION: nothing bounded how many fixtures ran `CREATE DATABASE` + the ~100-migration chain at once (measured peak: 45 concurrent test DBs / 19 blocked requests / 98 sessions on one container), pushing DDL past EF's default 30 s command timeout — every observed failure was `..ctor → MigrateAsync → SqlException 258 Execution Timeout Expired`, which xunit reports against every test in the class (hence "29 failures"). Fix: a machine-wide counting semaphore on `sp_getapplock` in `master` (`ScadaBridgeTestMigrate/0..N-1`, 4 slots, `SCADABRIDGE_MSSQL_TEST_MIGRATE_SLOTS` override) held across create+migrate — the arbiter must live on the shared SERVER because the contention is cross-process; gate connection is deliberately UNPOOLED (a pooled one defers the session-lock release until reuse, stranding a slot); crashed process ⇒ SQL releases the lock; a gate deadline miss proceeds UNGATED so the throttle can never itself be a red. Timeouts raised to 300 s migrate / 120 s create-drop (bounded queue = busy but progressing); once-per-process sweep of orphaned `ScadaBridgeAuditMigTest_*` >2 h; connectivity probe retries 3× (a busy server misread as absent flips every MSSQL test to a green-looking Skip — worse than a red). (2)+(3) Two more flakes behind it: `PartitionPurgeTests` and `NotifyDispatcherAuditTrailTests` LEAKED 100 ms-tick actors (purge / outbox dispatcher) whose only stop was timeout-bounded `TestKit.Dispose` — a leaked dispatcher even delivered the NEXT test's notification through its own exhausted adapter, stranding it `Retrying`. Fixed with `Watch`/`Stop`/`ExpectTerminated` in the creating test. Evidence: 10× concurrent ConfigurationDatabase.Tests went from 10/10 processes failing (17–77 failures each) to 3 consecutive 10/10-green runs; the AuditLog 3×+1× mix from failing to 3 clean runs; standalone AuditLog 383/383, ConfigurationDatabase 395/395 (no runtime regression), SiteCallAudit 66/66 (inherits the fixture); zero orphan DBs or stuck app locks left. Host.Tests/PlaywrightTests hit the PERSISTENT DB on an already-serialized axis — untouched. Systemic residue recorded as rows 95/96. Doc: `docs/test_infra/test_infra_db.md`. | Found independently by two agents in the 2026-08-16 wave | Closed 2026-08-17. | — |
| 83 | ~~**Post-boot start-path tests must gate on the DM's first instance-count report — `ReconcileStartsInstanceTests` still uses a bare `Task.Delay(500)`** *(Low, test-infra)*~~ **RESOLVED 2026-08-17 (`524bdbbe`).** `ReconcileStartsInstanceTests` converted to the first-`SetInstanceCounts`-report gate; the pattern was extracted into a shared `TestSupport/DeploymentManagerStartupProbe` now used by both this suite and `InstanceSweepSelfHealTests`, so the convention has one implementation to copy instead of two to keep in sync. SiteRuntime 633/633 green. | Row-78 fix session | Closed 2026-08-17. | — |
| 84 | ~~**`LogReplicationBacklogNote` can fire twice in one purge tick** *(Low, cosmetic)*~~ **RESOLVED 2026-08-17 (`ddb3c32e`).** The note is emitted at most once per tick. Fail-before: `RunPurge_WhenBothPhasesDeleteRows_LogsTheBacklogNoteOnceForTheWholeTick` fails pre-fix with `Assert.Single(): collection contained 2 matching items`. | Row-34 session | Closed 2026-08-17. | — |
| 85 | ~~**DCL connection actors are not spawned for connection configs that arrive via LocalDb replication — row 78's gap, one layer down** *(Medium-Low, availability; proven live 2026-08-16 by the row-78 close-out probe)*~~ **RESOLVED 2026-08-17 (`8f2c2813`) — with a premise correction AND a widened blast radius.** The row's mechanism was wrong: `data_connection_definitions` has NO production reader (its getter is test-only) — connection actors are created solely from the `Connections` map in an instance's own flattened config via `EnsureDclConnections`, so there was never a store→actor path to repair. The real defect: that ensure was the CALLER'S job and only 2 of 4 instance-start paths did it (boot batch ✅, `ApplyDeployment` ✅, `StartStoredInstance` ❌ — both the row-67 ensure-started and the row-78 sweep — and `HandleEnableResult` ❌; the two misses are exactly the two paths added later). So the bug was NOT replication-specific: an ordinary unreplicated site that boots with an instance disabled and enables it later hit the same dead end, because the boot load reads only enabled configs. Fix is structural: `EnsureDclConnections` moved INTO `CreateInstanceActor` (the documented single creation funnel), before the already-exists guard so the two working paths keep exact semantics; reuses the per-name hash guard so steady state is silent; a fifth path cannot forget. Store-level sweep pass explicitly REJECTED (memo §7.4): it wouldn't have fixed this (the table is empty on deploy-only sites) and the unguarded form ping-pongs the hash slot against the deploy path, dropping live subscriptions every tick. Fail-before by behavioral revert: 4 of 7 new tests fail (sweep-started, reconcile-ensure-started, enable-after-disabled-boot, exactly-once-across-sweeps). SiteRuntime 633/633, DCL 320/320. Sibling gaps found during the fix recorded as rows 89/90. | Proven live 2026-08-16 by the row-78 close-out probe (sweep started `rig-probe-1`/`soak-notify-1` but every tag subscribe failed for 6+ min with `Unknown connection: OPC PLC Simulator — will retry`; a second pair restart healed it) | Closed 2026-08-17. **Live-proven on the rig same day** in the exact original failure sequence (pair stopped, node-a `site-localdb.db` quarantined as `.probe85-quarantine-2026-08-17`, pair restarted): reconcile "0 fetched" at 08:29:59, then at the 08:30:54 sweep tick `Created DCL connection OPC PLC Simulator` fired ALONGSIDE the two self-heal Warnings (pre-fix they appeared alone), zero `No connection actor`/`Unknown connection` lines on either node, all three `rig-probe-1` tags Good with live values ~2 min after restart — NO second restart; steady-state silence confirmed (0 Created/Updated lines across subsequent ticks); replication reseeded (`localDbReplicationConnected: true`, backlog 0), site-b/c still correctly omit the fields. | — |
| 86 | **CLI and UI disagree on deployment site-scoping for Administrators** *(Medium, authorization)* | Found by the 2026-08-17 Low-tail sweep while re-verifying row 37 | `ManagementActor.cs:2586` grants an **Administrator role bypass** of site scoping on `deploy status`, while the UI's `SiteScopeService` is role-independent (system-wide iff the user carries zero `SiteId` claims). An Administrator carrying site claims therefore sees the whole fleet via the CLI but only their sites on the Deployments page — same user, same data, two answers, and one of them is wrong. This is row 37's duplication having drifted into behavior; decide which semantic is intended (role bypass vs claims-only) and enforce it on BOTH paths, ideally as part of the row-37 consolidation. | Next ManagementService/CLI session, or the row-37 consolidation — whichever comes first |
| 87 | **CLI deployment scoping is non-transactional and unbounded** *(Low)* | Found by the 2026-08-17 Low-tail sweep | The CLI path resolves site grants to an instance-id list in a separate, non-transactional read (`ManagementActor.cs:2590-2596`), reloading EVERY instance per scoped query; and `PageSize: 0` on the wire maps to `take: null` = "no limit" (`DeploymentManagerRepository.cs:434`), materializing the entire retention window in one query. The UI path is structurally incapable of both. Bounded today by the purge window and low CLI traffic. | Row-37 consolidation, or first slow/OOM `deploy status` report |
| 88 | **Unparseable CLI `--status` silently means "no filter"** *(Low, UX)* | Found by the 2026-08-17 Low-tail sweep | `ManagementActor.ParseDeploymentStatus` returns null on an unparseable status value, so a typo (`--status Sucess`) silently returns ALL statuses instead of an error — the caller cannot distinguish "filter applied, nothing matched" from "filter ignored". Fix: fail loudly on an unrecognized value (mirror the row-46 principle: loud failure over silent empty/full results). | Next CLI touch to the deploy command group |
| 89 | **Connection-only artifact deploy does not reach the peer node's DCL** *(Medium-Low, availability)* | Found by the 2026-08-17 row-85 fix session | `data_connection_definitions` IS one of the ten replicated tables, but only the node that received the deploy runs `HandleApplyArtifactDataConnectionsToDcl` — the peer gets the row via CDC and keeps serving the OLD endpoint until a restart or an instance redeploy touches it. Not fixable by another sweep: applying a definition change to a live connection needs an arbitration decision first (does the instance's flattened-config `Connections` hash or the standalone definition hash win when they disagree?), because the row-85 fix deliberately made the flattened config the sole creation source. | Next artifact-deploy or DCL session — decide the arbitration, then apply-on-CDC becomes safe |
| 90 | **No removal path for a connection deleted on the peer** *(Low, mirror image of row 89)* | Found by the 2026-08-17 row-85 fix session | `RemoveConnectionCommand` is sent only from the artifact-deploy reconcile on the node that processed the deploy; a connection deleted on the peer leaves a live connection actor here until restart. Bounded: the orphan actor serves a connection no instance references, wasting a device session but corrupting nothing. Same arbitration prerequisite as row 89 — fix them together. | With row 89 |
| 91 | **`SiteScriptCompileCacheTests.GetOrAdd_SameCodeAndGlobals_ComputesOnce` is parallel-load fragile** *(Low, test-infra; row-82 family)* | Found by the 2026-08-17 row-85 fix session: failed once in 4 full-suite runs, green in isolation AND green on base — pre-existing, probability merely raised by the suite growing | `SiteScriptCompileCache` is a process-wide static and the test asserts an absolute `Hits == 1` after a global `Clear()`; xUnit runs other classes in parallel in the same process, so any concurrently-compiling test breaks the absolute count. Recommended fix: row-74's established delta-over-scoped-action assertion pattern (a collection fixture would serialize more than needed). | Next SiteRuntime test touch, or first flake in a run that matters |
| 92 | **The event-log cap's size model is sample-based and blind to mid-history width shifts** *(Low, accuracy limitation — documented in memo §3.1)* | Row-81 fix session (2026-08-17) | Row width is sampled 500 rows from each END of `site_events`; a stream whose row width changes sharply mid-history is mis-sized until the trim reaches that region. Self-correcting per tick (each trim exposes new tail rows to the sample), and the model's deliberate +8–13% high bias absorbs modest shifts, but a pathological width profile could over- or under-trim for several ticks. | First field report of the cap trimming visibly wide of its target, or any rework of the size model (see rows 93/94) |
| 93 | **The cap model's page-layout constants are empirical for the CURRENT `site_events` schema** *(Low, maintenance trap)* | Row-81 fix session (2026-08-17) | Fill factors (0.78/0.75), cell overheads (16/8) and especially `IndexedColumnCount = 4` are calibrated to today's table + four indexes. A schema change — new column, new index — silently invalidates them; a fifth index in particular would be silently under-counted. Any `site_events` schema migration must re-run the memo §4 calibration and update the constants together. | Any schema change to `site_events` or its indexes |
| 94 | **`SqliteCapabilityTests` is a deliberate tripwire — do not "fix" it by weakening the assertion** *(Low, standing note)* | Row-81 fix session (2026-08-17) | The test FAILS BY DESIGN if a future SQLite bump ships `dbstat`/`sqlite_dbpage` support. The correct response at that point is to REPLACE the sampled page-layout model with exact `dbstat` measurement (retiring rows 92/93 with it), not to relax the test. The test's own comment says the same. | `SqliteCapabilityTests` fails after a SQLite/provider bump |
| 95 | **The leaked 100 ms-tick test-actor pattern is systemic — three more classes carry it** *(Low, test-infra; row-82 residue)* | Row-82 fix session (2026-08-17): two classes fixed (`PartitionPurgeTests`, `NotifyDispatcherAuditTrailTests`), three more found carrying the same shape | `AuditLogPurgeActorTests` (benign today only because exactly ONE test there touches the fixture DB — adding a second DB-backed test reintroduces row 82), `OutageReconciliationTests`, and `SiteAuditReconciliationActorTests` all create 100 ms-tick actors against a shared fixture DB with no deterministic stop (only timeout-bounded `TestKit.Dispose`, which does not fail on expiry). None failed across ~15 parallel runs, so left untouched for proportionate blast radius. Fix pattern is already written: `Watch`/`Stop`/`ExpectTerminated` in the creating test. | Next touch to any of the three classes, or first flake matching the leaked-actor fingerprint (a prior test's rows mutated mid-test) |
| 96 | **xunit parallelism is uncapped almost everywhere — `MaxParallelThreads` unset, only 3 of ~20 test projects have an `xunit.runner.json`** *(Low, test-infra policy)* | Row-82 fix session (2026-08-17) | Every suite runs at processor-count collection parallelism, and a solution-level `dotnet test` multiplies that across concurrent processes — the general multiplier behind the row-82/91 flake class. The row-82 fix bounds the DB-migration axis specifically; a project-wide parallelism default (or per-suite runner configs for the resource-heavy suites) is a policy decision worth making once, deliberately, rather than per-flake. Supporting evidence from the 2026-08-17 close-out run: `AuditLogPageTests.DrillInFromParentExecution_FiltersGridToSpawnerExecution` (Playwright, untouched by the wave) hit a 30 s navigation timeout under full-suite parallel load and passed isolated in 1 s — one more member of this class. | Next test-infra session, or CI adoption of parallel suite runs |
| 68 | ~~**`deploy.sh` run from a git worktree silently rehomes every node's bind-mount data dir**~~ **RESOLVED 2026-08-15 (guard added in the same close-out).** The compose files' bind mounts are relative (`./site-a-node-a/data`), so a deploy from a worktree checkout mounts the worktree's copies — the rig keeps running and looks healthy, but its SQLite stores split from the main checkout's and everything written there is orphaned by the next deploy from main. Bit the rows-46–53 close-out live: the seeding agent's `rig-probe-1` deploy persisted into worktree-homed dirs and "vanished" after the orchestrator's redeploy (no tombstone, no product bug — LWW/tombstone forensics ruled out deletion; the row sat in `.claude/worktrees/…/docker/site-a-node-a/data/site-localdb.db` the whole time). Both `docker/deploy.sh` and `docker-env2/deploy.sh` now refuse to run from a `.claude/worktrees/` path (override: `SCADABRIDGE_ALLOW_WORKTREE_DEPLOY=1`). | Found during the rows-46–53 close-out | Closed 2026-08-15. | — |
## Resolved (verified against the code 2026-07-10)
Rows removed from the Deferred table above once confirmed shipped. Kept here for traceability.
docs/plans/2026-07-21-auto-down-availability-decision.md
+18
View File
@@ -111,3 +111,21 @@ mis-explanation in the repo ("the alone-oldest is dead and cannot down itself").
- Monitoring already surfaces dual-active if it ever happens: both nodes report
`IsActive` in heartbeats / both `/health/active` = 200 — the Health dashboard shows
two Primaries.
## Postscript 2026-08-15 — the deprecation warning is benign (drill-verified), but pins us for Akka bumps
Every node logs at Cluster start: *"The `auto-down-unreachable-after` feature has been
deprecated as of Akka.NET v1.5.2 … The `keep-majority` split brain resolver will be used
instead."* Investigated and drill-verified on 1.5.62 (the same version this decision was
made and drilled on): the warning fires purely on the **presence of the key** and the
substitution does **NOT** happen when `downing-provider-class = "Akka.Cluster.AutoDowning"`
is set explicitly, as `BuildHocon` does. Proof by discriminating drill (`DRILL_MODE=active`,
2026-08-15): killing central-a — oldest AND lowest-address — left central-b to take over
(OldestChanged → all singletons moved, victim rejoined ready 2s after restart). Under
keep-majority, a 2-node even split keeps the lowest-address side, so central-b would have
downed ITSELF; it demonstrably did not. The decision stands exactly as written.
The real exposure is forward-looking: `AutoDowning` is deprecated and can be REMOVED by a
future Akka.NET release. **Any Akka version bump must re-run `DRILL_MODE=active
bash docker/failover-drill.sh` and re-read the release notes before being accepted**
(deferred-work register row 70).
docs/plans/2026-08-15-instance-sweep-self-heal-design.md
+335
View File
@@ -0,0 +1,335 @@
# Instance reconcile sweep — closing the config/runtime divergence gaps
**Date:** 2026-08-15 (§7 added 2026-08-17)
**Scope:** Site Runtime (`DeploymentManagerActor`), deferred-work-register rows **78**, **72**
and **85**
**Status:** implemented
---
## 1. The problem
The site's `deployed_configurations` rows are the durable truth about what should be running.
The set of live Instance Actors is what *is* running. Nothing kept the two in agreement after
boot, and there was no signal when they disagreed.
Before this change, an Instance Actor could be created from exactly three places:
1. the boot-time SQLite load in `DeploymentManagerActor.PreStart`;
2. an explicit deploy / enable command from central;
3. (register row 67, commit `482165cb`) `SiteReconciliationActor`'s `EnsureInstanceStartedCommand`,
raised for enabled instances **the reconcile pass itself fetched and stored**.
### Row 78 — configs that arrive by replication start nothing
*(Medium-Low, availability; proven live 2026-08-15.)*
`deployed_configurations` is one of LocalDb's ten replicated tables. On a replicated site pair,
a config can therefore appear in a node's file with **no local actor involved at all**: the CDC
sync applier writes the row directly.
The live sequence was:
- site-a-b booted solo; its startup reconcile fetched 4 configs from central and stored them
(row 67's machinery started their actors on that node — correct).
- The pair was then restarted with a **fresh** site-a-a file. CDC snapshot replication seeded
site-a-a's `deployed_configurations` within ~5 s of boot.
- That was **after** site-a-a's boot-time load had already read an empty table, and **invisible**
to site-a-a's own reconcile pass: central rightly reported *no gap* ("0 fetched"), because
central's view of the site was correct — the site really did have the configs.
- Result: enabled instances sat dead with their configs present (`debug snapshot` →
`instanceNotFound`) until yet another restart.
Row 67 covers configs the reconcile *fetches*. This is the same gap through a side door.
### Row 72 — a refreshed config does not reach the running actor
*(Medium-Low, same silent-divergence class.)*
`ReconcileGapItem` covers both *missing* and *stale* configs. For a **stale** item the reconcile
writes the newer config via `StoreDeployedConfigIfNewerAsync` — a pure file write. The running
Instance Actor keeps executing the **previously compiled** config until the next restart or
redeploy, and nothing anywhere surfaced "stored revision ≠ running revision".
Row 72 was deliberately deferred out of row 67's scope because restarting a *live* instance
carries redeploy semantics that needed their own decision. That decision is section 3.
---
## 2. The mechanism: one periodic sweep in the Deployment Manager singleton
A single periodic tick on the Deployment Manager singleton re-reads every durable
`deployed_configurations` row and repairs both divergences.
```
InstanceSweepTick (every InstanceSweepIntervalSeconds, default 60)
└── off-thread read of all deployed configs
└── InstanceSweepLoaded → actor thread
for each ENABLED stored config, unless a deploy pipeline owns the instance:
no running actor → start it (row 78)
running, revision ≠ stored → re-apply stored (row 72)
```
**Ownership.** The Deployment Manager is a cluster singleton scoped to the site-specific role, so
the sweep runs on exactly one node — the active one — and Instance Actors stay under the
singleton, which is the same invariant row 67's singleton-proxy routing preserved.
### 2.1 Why a sweep and not a LocalDb change hook
The register suggested either. A sweep was chosen:
- **Coverage.** A replication-apply hook covers exactly one arrival path, and only on the half of
the fleet that runs with replication enabled (rig posture: site-a replicated, site-b/site-c
deliberately not). The sweep is arrival-path-agnostic — replication seeding, a post-boot
snapshot resync, an operator restoring a file by hand, and any future path all reduce to the
same question: *does the store agree with the runtime?*
- **Availability.** `ILocalDb` exposes **no** change-notification, applied-rows callback or
observer API — verified against `ZB.MOM.WW.LocalDb` 0.3.0. A hook would require a new upstream
capability in the `scadaproj` library plus an in-process bus from the sync applier into this
singleton, i.e. a cross-repo change to close a site-local gap.
- **Reuse.** The sweep inherits row 67's guards verbatim; the shared `StartStoredInstance` helper
means the reconcile heal and the sweep heal produce identical bookkeeping and identical
operator-facing events, differing only in the trigger they name.
Cost is one SQLite read per minute per site — the same read `HandleDeploymentStateQuery` already
performs per central query.
### 2.2 Guards (all inherited or shared with row 67)
- **Disabled instances are never started.** A disabled instance legitimately has a config row and
no actor; that is its correct steady state.
- **The sweep defers to any deploy pipeline that owns the instance** — a compile warm in flight
(`_deployWarms`), a predecessor still terminating (`_terminatingActorsByName`), or a deployment
awaiting its persistence/init join (`_inFlightDeployments`). This is the same
`InstanceIsOwnedByADeployPipeline` predicate row 67's `ShouldAttemptEnsureStart` is now built
from. It matters: a redeploy passes through a window where the predecessor has been stopped and
removed from routing but the replacement does not exist yet, which an unguarded sweep would read
as "enabled config, no actor" and answer by creating a child whose actor name is still
registered.
- **Inert until boot-time creation finishes.** A tick landing between two staggered startup
batches would "heal" instances the boot load is already about to create, announcing a fault on
a node that is merely still starting.
- **One repair attempt per (instance, revision) pair.** This is what keeps a *periodic* self-heal
from becoming a restart loop: an Instance Actor that dies during init leaves its config row in
place, and a stored config the site compile gate rejects can never apply. Without the claim,
either would be re-attempted — and re-announced — every tick forever. A genuinely new revision
is a different key and is always attempted.
- **No overlapping reads.** A slow storage read cannot let ticks stack up into duplicate repairs.
- **Silent when healthy.** A sweep that finds nothing emits no log line above Trace and no site
event. This runs on every tick of every healthy site, forever.
---
## 3. The row-72 decision: converge, do not merely warn
**Decision.** When the sweep finds a running instance whose revision no longer matches the stored
row, it **applies the stored config through the same per-instance apply/restart path a central
redeploy uses** (`HandleDeploy` → compile gate → stop/watch/recreate → persist), and announces
both revisions in a Warning `deployment` site event.
**Rationale.**
- The stored row **is** what central successfully deployed. The running actor is executing a
config central believes was replaced. Leaving it running is the actual fault — a site quietly
serving superseded logic is worse than a controlled restart, and the operator has no way to
discover it.
- The alternative (surface-only) leaves an operator with a Warning they can only act on by
issuing a redeploy — the exact operation the sweep would have performed, minus the delay and
minus the chance they see it.
- Reusing `HandleDeploy` rather than writing a second apply path means the compile gate, the
warm-then-gate ordering, the redeploy-supersede semantics, the static-override and
native-alarm reset, and all-or-nothing-per-instance are the ones already in production. No new
semantics were invented. In particular a stop still does **not** cancel an in-flight script
run, exactly as for a central redeploy.
**Why this is safe against the ordering hazards the register worried about.**
- The sweep never acts on an instance a deploy pipeline owns, so it cannot interleave with a real
redeploy; and if a deploy arrives *after* the sweep's own deploy is queued, it is ordinary
last-write-wins on the same mailbox, which is a case the deploy path already handles.
- A rejected restart is inert, not partial: `RunDeployGateAndProceed` returns **before** anything
is stopped, so the running instance keeps serving its old config. It is announced once
(Warning for the attempt, Error for the failure) and never retried for that revision.
- The synthetic `DeployInstanceCommand` replies to `Self`, not to `NoSender` — dead letters are a
monitored site health metric, and a self-heal must not manufacture them.
**Row 72's "nothing surfaces it" is false either way.** Even in the failure path the divergence
produces two operator-visible `deployment` site events naming both revisions.
---
## 4. Configuration
`ScadaBridge:SiteRuntime:InstanceSweepIntervalSeconds` — default **60**, validated `>= 0`
(`SiteRuntimeOptionsValidator`). **`0` disables the sweep**, which reopens both gaps; the row-67
`EnsureInstanceStartedCommand` path is unaffected either way. No config change is required on any
node to get the fix — the default is live.
## 5. Observability
| Condition | Log | Site event |
|---|---|---|
| Nothing to repair | — | — |
| Row 78 start | Warning naming the instance and trigger | `deployment` / **Warning** — "started by deployment-manager sweep self-heal" |
| Row 72 restart attempt | Warning naming both revisions | `deployment` / **Warning** — "restarted by deployment-manager sweep self-heal — running revision X is stale, applying stored revision Y" |
| Row 72 restart applied | Information | — (the deploy path's own Info event covers it) |
| Row 72 restart rejected | Warning with the gate's errors | `deployment` / **Error** — "stale-revision deployment-manager sweep self-heal failed" |
| Storage read failed | Warning | — |
The trigger is named in the message so a sweep heal is distinguishable from row 67's
"startup reconcile self-heal".
## 6. Tests
`tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceSweepSelfHealTests.cs` — 10 tests.
Fail-before evidence (behavioral revert: the periodic timer left unarmed) — **5 fail, 5 pass**:
| Test | Pre-fix |
|---|---|
| `Sweep_StartsEnabledInstance_WhoseConfigArrivedWithoutADeployOrReconcile` | **FAIL** — the live row-78 symptom |
| `SweepStartedInstance_EmitsAWarningSiteEvent_NamingTheSweepTrigger` | **FAIL** |
| `Sweep_StartsTheInstanceExactlyOnce_AcrossManySweeps` | **FAIL** |
| `Sweep_RestartsInstance_ServingAStaleRevision` | **FAIL** — the row-72 symptom |
| `Sweep_StaleRestartRejectedByTheCompileGate_IsAnnouncedOnceAndNotRetried` | **FAIL** |
| `Sweep_NeverStarts_ADisabledInstance` | pass (negative guard) |
| `Sweep_DefersToAnInFlightRedeploy` | pass (negative guard) |
| `Sweep_DoesNotRestart_WhenStoredAndRunningRevisionsMatch` | pass (negative guard) |
| `Sweep_IsSilent_WhenThereIsNothingToHeal` | pass (negative guard) |
| `Sweep_DoesNothing_WhenTheIntervalIsZero` | pass (negative guard) |
Every test gates on the Deployment Manager's first instance-count report before writing a config
row, because a row written before the boot-time load has read storage is picked up by that load —
which is precisely the path row 78 is about *not* having. Without the gate a test would silently
prove the boot load works.
That gate is now a shared helper,
`tests/…/TestSupport/DeploymentManagerStartupProbe.cs`, and **register row 83** applied it to
row 67's suite (`ReconcileStartsInstanceTests`), which had the same trap behind a bare
`Task.Delay(500)`. Those tests also pin `InstanceSweepIntervalSeconds = 0`: with the sweep armed
it would independently start the same instances, and the suite would stop distinguishing the
row-67 mechanism from the row-78 one. The remaining `Task.Delay`s in that file are deliberate —
they precede assertions that *nothing happened*, which cannot be awaited on.
---
## 7. Row 85 — a self-healed instance started with no DCL connection actors
*(Medium-Low, availability; proven live 2026-08-16 by the row-78 close-out probe. Added
2026-08-17.)*
Row 78 got the Instance Actor running. Row 85 is what was still missing underneath it: the
instance came up with **no data connection actors**, so every tag it subscribed was answered
`Unknown connection: OPC PLC Simulator` by the DCL manager — for 6+ minutes, until a second pair
restart. Same class as row 78, one layer down.
### 7.1 Where connection actors actually come from
The register's suggested fix was "ensure a connection actor exists for every stored site data
connection". Reading the code first changed the shape of the answer, because the premise is not
how the runtime works:
- **`data_connection_definitions` has no production reader.**
`SiteStorageService.GetAllDataConnectionDefinitionsAsync` is called from exactly one place in
the repo, and it is a test. Nothing loads that table at boot.
- A DCL connection actor is created **only as a side effect of starting an instance**, from the
`Connections` map inside that instance's *own flattened config*, via `EnsureDclConnections`
(plus the separate system-wide artifact-deploy path, `HandleApplyArtifactDataConnectionsToDcl`).
So "the connection row sat in the store with no actor" was right about the symptom and wrong
about the mechanism: there is no store→actor path to repair. The real defect is that
`EnsureDclConnections` was **the caller's job, and only two of the four callers did it**:
| Path | Creates the Instance Actor | Ensured its connections (pre-fix) |
|---|---|---|
| Boot-time staggered batch | `CreateInstanceActorBatch` | ✅ |
| Central deploy / redeploy | `ApplyDeployment` | ✅ |
| **Self-heal start** (row 67 ensure-started **and** row 78 sweep) | `StartStoredInstance` | ❌ |
| **Enable** a disabled instance | `HandleEnableResult` | ❌ |
The two misses are exactly the two paths that were added *after* the original pair, which is the
signature of a convention that lives at call sites instead of in the funnel.
**The enable path proves this was never replication-specific.** The boot-time load reads only
ENABLED configs, so a node that boots with an instance disabled never creates that instance's
connections; enabling it afterwards hit the identical dead end on an ordinary, unreplicated
site. Row 85 was reachable with no LocalDb replication anywhere in the picture — the replication
sequence is just how it was found.
### 7.2 The fix: move the ensure into the single creation funnel
`CreateInstanceActor` is already documented as *"the single creation funnel, so recording it here
covers every path that can ever start an Instance Actor"* — it is where `_runningRevisions` is
stamped for exactly that reason. `EnsureDclConnections(configJson)` now sits at the top of it,
and the two caller-side calls are removed.
- **Placed BEFORE the already-exists guard**, so the two paths that already called it keep their
exact previous semantics (both called it unconditionally, before `CreateInstanceActor`).
- **No new message, no new option, no DCL change.** It reuses `EnsureDclConnection`'s existing
`_createdConnections` hash guard, so an unchanged connection is a no-op — no message, no log,
nothing on the wire. Steady state is silent, which matters because this now runs on every
instance start on every node.
- **Structural, not another guard.** A fifth start path cannot forget, because there is nothing
left to remember.
### 7.3 Why the DM and not the DCL manager
The register offered either. The DCL manager was rejected: it has no storage handle and no notion
of "the connections this site ought to have" — it is a router over actors it was told to create.
Teaching it to read `data_connection_definitions` would give the DCL a **second, divergent config
source** alongside the flattened instance configs the DM already pushes, with no arbiter when the
two disagree. The DM owns `_createdConnections`, owns the flattened configs, and is already the
only sender of `CreateConnectionCommand`/`RemoveConnectionCommand` — it is a single funnel
already, and the fix keeps it that way.
### 7.4 Why NOT a store-level connection pass in the sweep
A sweep pass over `data_connection_definitions` was designed and **deliberately dropped**. It is
recorded here because it looks obviously right and is not:
- **It would not have fixed row 85.** That table is written only by the artifact-deploy path, so
on a site that has only ever received instance deploys it is empty — while the connections that
matter live in the instances' flattened configs.
- **The unguarded version is actively harmful.** `_createdConnections` holds one hash per name.
The flattened-config hash and the artifact-definition hash for the same connection can
legitimately differ (artifact newer than an instance's flattened config — the supported
version-skew case). A sweep re-hashing from the definitions table would flip the slot every
tick against the deploy path flipping it back, **recreating the connection actor and dropping
its live subscriptions on a loop**.
- **The guarded version is near-empty.** Restricting it to names absent from `_createdConnections`
removes the ping-pong but then covers only "a definition row exists that no instance
references" — where no connection actor is wanted anyway.
**Residual gap, accepted and recorded:** `data_connection_definitions` *is* one of LocalDb's ten
replicated tables, so a **connection-only** artifact deploy replicates to the peer node, and only
the node that received the deploy runs `HandleApplyArtifactDataConnectionsToDcl` — the peer keeps
serving the old endpoint until something restarts it or an instance is redeployed. That is a real
sibling of row 85, it is **not** closed here, and closing it needs the arbitration decision above
(which source wins), not another sweep pass. Flagged for the deferred-work register.
**Removal side:** also unchanged and out of scope. `RemoveConnectionCommand` is sent only from the
artifact-deploy reconcile; a connection deleted on the peer via replication leaves a live actor on
this node. Same root cause, same arbitration question, same register entry.
### 7.5 Tests
`tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SelfHealCreatesDclConnectionTests.cs` —
7 tests, asserting on the `CreateConnectionCommand` messages the DM sends to a recording stand-in
for the DCL manager (the message whose absence *was* row 85).
Fail-before evidence (behavioral revert: `EnsureDclConnections` removed from the funnel and
restored at the two original call sites — i.e. the exact pre-fix code) — **4 fail, 3 pass**:
| Test | Pre-fix |
|---|---|
| `SweepStartedInstance_AlsoCreatesItsDclConnection` | **FAIL** — the live row-85 symptom |
| `ReconcileEnsureStartedInstance_AlsoCreatesItsDclConnection` | **FAIL** — row-67 path |
| `EnabledInstance_DisabledAtBoot_AlsoCreatesItsDclConnection` | **FAIL** — no replication involved |
| `SweepStartedInstance_CreatesTheConnection_ExactlyOnce_AcrossManySweeps` | **FAIL** (0, not 1) |
| `Sweep_SendsNoConnectionCommands_WhenThereIsNothingToHeal` | pass (steady-state silence) |
| `OrdinaryDeploy_StillCreatesItsDclConnection` | pass (refactor guard) |
| `BootLoadedInstance_StillCreatesItsDclConnection` | pass (refactor guard) |
The two refactor guards are the point of passing both before and after: they pin that moving the
call out of `CreateInstanceActorBatch` and `ApplyDeployment` did not regress the paths that
already worked.
docs/plans/2026-08-15-loadharness-results-1h.json
+4675
View File
File diff suppressed because it is too large Load Diff
docs/plans/2026-08-15-site-events-purge-burst-measurement.md
+226
View File
@@ -0,0 +1,226 @@
# `site_events` purge burst — measurement + pacing decision (register row 34)
**Date:** 2026-08-15
**Row:** deferred-work register #34 — *"`site_events` purge burst is unbounded in the oplog and unmeasured at scale."*
**Outcome:** measured; **pacing implemented** (a per-tick delete ceiling with an accelerated
catch-up tick). One separate defect found on the storage-cap path — see *New issue* at the end.
---
## 1. What the row asked
`site_events` lives in the consolidated LocalDb file and is CDC-replicated on a replication-enabled
site. The daily retention purge (30 days) and the storage-cap trim (1 GB) slice their deletes at
1000 rows per DELETE statement, but the **whole** expired set went in one tick, so the oplog gained
one row per deleted event with no pacing. The wire path is bounded (`MaxBatchBytes` 2 MB; the rig
also pins `MaxBatchSize: 64` to bound the DB read page) but the oplog **write** was not, and nothing
had been measured.
The design commitment that had to survive, and did: **there is no purge-exemption path.** Purge
deletes are captured like any other write — the same property that made `ReplaceAllAsync` unsafe to
reintroduce. Nothing below adds an exemption or an oplog bypass; the fix bounds how much one tick
deletes, not what gets captured.
## 2. How it was measured
A throwaway console harness (`dotnet` console app, scratchpad — not committed) running the **real**
`EventLogPurgeService` over a **real** `ZB.MOM.WW.LocalDb` file with `site_events` genuinely
`RegisterReplicated`, plus a two-node loopback replication pair for the drain leg:
| Phase | What it does |
| --- | --- |
| `calibrate` | Seeds realistically-shaped rows, measures `page_count`/`freelist_count`/`page_size` delta ⇒ bytes per row ⇒ rows at the 1 GB cap. |
| `burst` | Seeds N expired rows, registers CDC, parks the oplog at "everything acked", runs one real `RunPurge()`. Records purge wall time, oplog rows, oplog payload bytes, row-version tombstones, file size before/after. |
| `cap` | Same but with rows *inside* retention and a small `MaxStorageMb`, so only the storage-cap trim runs. Run twice — replicated and not — to isolate the CDC contribution. |
| `drain` | Two full replication stacks (initiator + passive `MapZbLocalDbSync`) over Kestrel h2c on loopback, both nodes registered for `site_events`. Seeds through the normal write path so the pair converges naturally, then purges on node A and polls the O(1) backlog expression until it returns to 0 and node B's table is empty. |
Row content mirrors the post-WP3.2 event stream: mostly short Info rows, 1-in-10 carrying a details
payload, instance ids on 2-in-3. `docker/` was not touched.
Machine: macOS (darwin 25.5.0), .NET 10, Release build, server GC, local SSD. Loopback drain numbers
are therefore an **upper bound on throughput** — a real inter-node link is slower, and the margins
below are stated with that in mind.
## 3. Numbers
### 3.1 How big the worst case actually is
```
calibrate: 200,000 rows
bytes per row (incl. the four indexes) 393
rows at the 1 GB storage cap 2,731,947
```
The 1 GB cap is the real bound on a single retention burst: the worst realistic case is a log at the
cap in which everything has expired (a site restarted after a long outage, a clock correction, or
simply the first purge after `site_events` retention was ever satisfied).
### 3.2 The unpaced retention burst
| Seeded expired rows | Purge wall time | Oplog rows queued | Oplog payload | Row-version tombstones | File before → after |
| ---: | ---: | ---: | ---: | ---: | --- |
| 500,000 | 36.2 s | 500,000 | 52.0 MB | 500,000 | 254 MB → 129 MB |
| **2,731,947** (1 GB cap) | **285.1 s** | **2,731,947** | **284.1 MB** | **2,731,947** | 1.39 GB → 707 MB |
**The headline finding:** the worst-case burst *queues* **2.7x LocalDb's `MaxOplogRows` default of
1,000,000**. Crossing that ceiling is not cosmetic — `OplogStore.EnforceCapsAsync`, which the
replication maintenance service runs every 60 s, prunes the oldest oplog rows **and sets
`needs_snapshot = 1`**, the database-wide flag. The next handshake therefore serves a **full
snapshot of all ten registered tables**. A routine retention purge could, on its own, force a full
resync of the site's entire replicated state.
**How much of that 2.73 M is actually resident depends on the peer** — the oplog is drained
concurrently, so the *peak backlog* is the burst minus what the peer acks while the purge runs. §3.3
measures it: on a loopback pair the peak was 391 k, not 2.73 M, because the peer kept up. That does
not make the risk theoretical, it makes it **conditional on peer speed**, which is exactly the
property a design should not depend on:
- **peer down / disconnected** — nothing drains, the full 2.73 M is resident. Over the cap outright.
- **peer slower than ~63 % of loopback** — the production deficit outruns the drain and the peak
crosses 1 M. Arithmetic from §3.3: the purge produced 8,548 rows/s and the loopback peer consumed
7,585 rows/s; halve the peer's rate and the peak lands ≈1.5 M.
- **peer at loopback speed** — peak 391 k, under the cap, but still 391x the row-33 badge threshold.
Pacing removes the dependency entirely: a 250 k tick cannot exceed the cap at *any* peer speed,
including a peer that is down.
Secondary costs at worst case: 285 s of purge (the write lock is taken and released 2,732 times, so
recorder flushes interleave — but they contend for nearly five minutes), 284 MB of oplog payload,
and 2.73 M row-version tombstones retained for `TombstoneRetention` (7 days). The file was still
707 MB *after* deleting a 1 GB event log, because the replication bookkeeping the delete generated
is itself in that file.
### 3.3 Drain
Two-node loopback pair, `MaxBatchSize: 64` (the rig's pin), `MaxBatchBytes` at its 2 MB default.
Both nodes registered for `site_events`; rows are seeded through the normal write path so the pair
converges naturally before the purge runs.
| Rows | Insert leg (seed + replicate) | Purge | Backlog → 0 | **Peak backlog** | Tombstones/s | Inside the 15-min dwell? |
| ---: | ---: | ---: | ---: | ---: | ---: | --- |
| 20,000 | 1.3 s (15.0 k/s) | 0.5 s | 1.0 s | 17,424 | 19,807 | yes |
| 500,000 | 65.7 s (7.6 k/s) | 34.4 s | 46.3 s | 150,912 | 10,795 | yes |
| **2,731,947** (1 GB cap) | 996.9 s (2.7 k/s) | 319.6 s | **360.2 s** | **391,424** | 7,585 | yes (6.0 min) |
**Drain is not the bottleneck, and a healthy peer blunts the burst.** At every size the pump
consumed tombstones almost as fast as the purge produced them, so the resident backlog stayed far
below the burst size — 391 k peak on a 2.73 M burst. The tombstone leg is roughly 4x cheaper per row
than the insert leg, as expected: a delete oplog entry carries a null `row_json`.
**That is exactly why the unpaced burst was still unsafe**, though. The peak is
`produced − drained`, so it is a function of the *peer's* throughput, and the margin evaporates as
soon as the peer is slower than loopback or absent (see §3.2). A design whose oplog-cap safety
depends on the peer keeping up is one bad link away from a full ten-table resync.
**Row-33 interplay (the sustained badge).** The badge needs the backlog above 1000 continuously for
15 minutes.
- **Paced (shipped default).** A full 250 000-row tick drains in ~23 s at the measured rate; even
assuming a real link is **10x slower** than loopback it clears in ~4 minutes — comfortably inside
the dwell, with the next tick 5 minutes away. Pacing lowers each spike and lengthens *total*
purge duration, but every individual spike still drains between ticks, so the badge assumption
holds with a large margin. **Verified, not assumed:** the 500 k measurement is 2x a paced tick and
drained in 46 s.
- **Unpaced (previous behaviour).** 6.0 minutes on loopback — inside the dwell, but only 2.5x of
margin, at a peak of 391 k (391x the badge threshold). At a 3x-slower link the same burst badges.
So pacing does not merely preserve the row-33 assumption; it is what makes it robust.
### 3.4 The storage-cap trim
`MaxStorageMb = 64`, 400,000 rows seeded inside the retention window (so only the cap path runs):
| Replicated? | Events deleted | Remaining | Measured size before → after | Reached the cap? |
| --- | ---: | ---: | --- | --- |
| no | 248,000 | 152,000 | 158.0 MB → 66.98 MB | **yes** (cap 67.11 MB) |
| yes | 174,000 | 226,000 | 203.6 MB → 167.7 MB | **no — stopped 2.5x over cap** |
See *New issue* below.
## 4. Decision — pace it
Pacing is warranted, on the strength of §3.2: an unbounded tick can exceed `MaxOplogRows` and buy a
full ten-table snapshot resync. Drain *speed* (§3.3) was not the reason — that is healthy — but the
fact that the safety margin **depends on** drain speed is: with the peer down, the whole 2.73 M
burst is resident and the cap is breached outright. Bounding the tick makes the outcome independent
of the peer, which is the property the previous design lacked.
**Shipped:**
- `SiteEventLogOptions.MaxPurgedRowsPerTick`, default **250 000**, `0` = unlimited.
- Ceiling on rows deleted by **one tick**, **shared** between the retention purge and the cap
trim. Two independently-capped phases would let a tick emit twice the intended burst.
- `0` (not a negative, which the validator rejects) is the documented escape hatch back to the
pre-row-34 behaviour, byte-for-byte.
- **Why 250 000:** a 4x margin under `MaxOplogRows` = 1 000 000, and that margin is shared with
every *other* replicated table's ordinary traffic, which also occupies the same oplog. At the
measured rates one such tick costs ~17 s of purge and ~23 s of drain.
- `SiteEventLogOptions.PurgeCatchUpInterval`, default **5 minutes**.
- A tick that stopped at the ceiling schedules the next tick after this instead of
`PurgeInterval`. Without it, pacing would make retention converge in 11 **days** at the daily
cadence — a policy that no longer enforces its own window. With it, the worst case converges in
~11 ticks ≈ **55 minutes**, every burst bounded.
- **Why 5 minutes rather than 1:** the interval must exceed the peer's drain time for one tick, or
ticks outrun the drain and the backlog accumulates across them — reintroducing the very problem
being fixed. 250 k drains in ~23 s on loopback; 5 minutes leaves ~13x headroom for a slower
real link.
- Validated `> 0` and `<= PurgeInterval` (a longer catch-up would *delay* convergence, inverting
its purpose).
- **The ceiling is not claimed when the work is finished.** An expired set that lands exactly on the
budget is done; the retention phase runs a bounded `EXISTS` probe before flagging a catch-up tick,
so a converged node never arms a 5-minute retry that deletes nothing.
- **Visibility.** A ceiling-limited tick logs a Warning naming the ceiling, the rows deleted and the
accelerated interval — **rate-limited to one line per 15 minutes**, because a large backlog
produces a run of such ticks. Every hit increments
`scadabridge.site.event_log.purge_ceiling_hits` (`ScadaBridgeTelemetry`, whose meter is already in
the site `ObservedMeters` allowlist). The counter is the unthrottled signal, and its **shape** is
the diagnosis: a run that stops = a backlog converging; hits that never stop = the site writes
events faster than one paced tick purges, so the ceiling or the catch-up interval is mis-sized.
- **Log wording updated.** The existing replication-backlog Information note now says the spike is
bounded by the ceiling rather than by how much history expired.
**Consequence accepted:** the purge loop is now a `Task.Delay` loop rather than a `PeriodicTimer`,
because the wait is no longer a constant. The cadence is therefore anchored to the **end** of the
previous purge rather than to process start. At a daily interval and a measured worst-case paced
tick of ~17 s the drift is immaterial; the topology guide and component doc say so.
**Explicitly not done:** no purge-exemption path, no oplog bypass, no "skip capture for deletes"
flag. Those are ruled out by design and the measurement gives no reason to revisit that.
## 5. New issue found — the storage cap cannot be met on a replicated node with an undrained oplog
Not row 34, and not fixed here. `EventLogPurgeService.GetDatabaseSizeBytes()` measures
`PRAGMA page_count - freelist_count` — i.e. the **whole consolidated LocalDb file**, which since
Phase 1/2 holds ten tables *plus* `__localdb_oplog` and `__localdb_row_version`. On a replicated
node the cap trim therefore fights itself: each 1000-row delete frees ~390 KB of `site_events` pages
and adds ~1000 oplog rows plus ~1000 row-version tombstones to the same file. Measured (§3.4) the
trim stopped **2.5x over a 64 MB cap** with 226 k rows still present, tripped by the existing
"size did not decrease across an iteration" guard — which is doing its job (it prevents the table
being wiped) but leaves the cap unenforced.
Severity is moderate, not acute: with a live peer the oplog drains on ack and successive ticks
recover ground, and tombstones expire after 7 days. But two real consequences follow — a node whose
peer is down cannot enforce its event-log storage cap at all, and the "1 GB event log" budget is in
practice shared with replication bookkeeping that the event log does not own. A correct fix would
measure `site_events`' own pages (e.g. `dbstat`) rather than the file, which is a change to the
cap's semantics and deserves its own decision. Reported for the register; no row added here.
> **Postscript (2026-08-17) — fixed; register row 81 closed.** The cap now measures `site_events`
> and its four indexes only. One correction to the suggestion above: **`dbstat` turned out to be
> unavailable.** It is a compile-time option and the bundled `SQLitePCLRaw.lib.e_sqlite3`
> (SQLite 3.53.3) does not enable it — `PRAGMA compile_options` lists no `ENABLE_DBSTAT_VTAB`, and
> neither `dbstat` nor `sqlite_dbpage` appears in `pragma_module_list`. The table's footprint is
> therefore *modelled* from sampled row width rather than measured, at −0.8 %…+12.8 % accuracy
> against the numbers in §3.4, biased high on purpose. The "size did not decrease" guard survives,
> re-based onto the table-scoped metric so replication bookkeeping can no longer trip it. Decision
> record, including the probe evidence and why a different SQLite build was rejected:
> `docs/plans/2026-08-17-site-event-log-cap-semantics-decision.md`. §3.4's replicated row is left
> as measured — it is the record of the defect, not of current behaviour.
## 6. Verification
- Fail-before: the new behavioural tests do not compile against pre-fix `EventLogPurgeService`
(`RunPurge` was `void`); with the fix restored but `PurgeBudget` neutralised to always-unlimited,
**6 of the new tests fail** and the 19 pre-existing ones pass — so the assertions are carried by
the ceiling, not incidentally.
- `dotnet build ZB.MOM.WW.ScadaBridge.slnx` clean.
- `ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests`: 93 passed / 0 failed.
docs/plans/2026-08-15-target-scale-load-test-results.md
+18
View File
@@ -162,6 +162,24 @@ flattens once gen-2 collections begin, or issue an explicit
`GC.Collect(2, Forced, blocking: true)` at window start and end and compare.
Deliberately *not* attempted here — the brief was to measure, not to tune.
**SETTLED 2026-08-15 — the 1-hour protocol was run (register row 29): NOT a leak.**
Same harness, same scale, `--sustain-minutes 60` (3,911 s total, 135,041,250 updates
offered at 37,511/s = 100.0 % of nominal, on the same host, concurrent with the docker
rig and a central failover drill). The decisive numbers over the 3,600 s steady-state
window (361 samples): **working set 2,722 → 1,757 MB, slope −28.89 MB/min** (peak
3,258 MB), **one gen-2 collection observed**, threads flat at 59. A working set that
*falls* by ~1 GB across a full sustained hour is incompatible with a slow leak; the
managed-heap least-squares slope (+16.26 MB/min, end 3,352 MB, peak 3,726 MB) remains
an artifact of measuring across sawtooth segments between rare gen-2 collects, exactly
as F1 hypothesised. Every other metric held or improved on the 20-minute run: tag
latency P50 0.92 ms / P99 4.01 ms / max 92.18 ms across 2.9 M samples, **0 events
dropped** on live subscribers, health-report ingest P99 0.05 ms (1,240/1,240, 10 sites),
debug snapshots 744/744 with 0 timeouts, S&F 20,000 msgs buffered at 16,837 msg/s and
drained at 3,418 msg/s to depth 0 (same one-`DefaultRetryInterval` pre-drain wait, 29.0 s
— F2 unchanged), slow-subscriber isolation structural and total (4 healthy peers at
100.00 % while the stalled probe lost 98.81 % entirely in its own channel — F3
unchanged). CPU 2.9 % of the box.
### F2 — Store-and-forward waits one full `DefaultRetryInterval` before draining a deferred backlog (informational, by design)
The drain measurement initially read as **33 msg/s**, which would have been alarming.
docs/plans/2026-08-17-site-event-log-cap-semantics-decision.md
+264
View File
@@ -0,0 +1,264 @@
# `site_events` storage cap — what the cap measures (register row 81)
**Date:** 2026-08-17
**Row:** deferred-work register #81 — *"the storage cap fights its own oplog."*
**Outcome:** cap semantics changed. `ScadaBridge:SiteEventLog:MaxStorageMb` now bounds
**`site_events` and its four indexes**, not the whole consolidated LocalDb file. `dbstat` — the
exact measurement the register proposed — turned out to be **unavailable in the SQLite this
solution ships**, so the table's footprint is *modelled* from sampled row width. This memo records
the probe that ruled `dbstat` out, the model, its measured accuracy, and the new meaning of the
wipe-prevention guard.
---
## 1. The defect
Since LocalDb Phase 1/2 the site keeps ten tables in one SQLite file at `LocalDb:Path`, alongside
LocalDb's own `__localdb_oplog`, `__localdb_row_version` and `__localdb_snapshot_state`
bookkeeping. `EventLogPurgeService.GetDatabaseSizeBytes()` measured
`page_count − freelist_count` over **that whole file** and compared it to `MaxStorageMb`.
On a replication-enabled node the cap trim therefore fought itself. Every 1000-row batch freed
~390 KB of `site_events` pages *and* pushed ~1000 rows into `__localdb_oplog` plus ~1000
row-version tombstones into the same file. The measured size did not fall, the existing
"size did not decrease across an iteration" guard fired, and the trim stopped — with the cap
unmet.
Measured 2026-08-16 during the row-34 work
(`docs/plans/2026-08-15-site-events-purge-burst-measurement.md` §3.4/§5), 64 MB cap, 400 000 rows:
| Replicated? | Events deleted | Remaining | Size before → after | Reached the cap? |
| --- | ---: | ---: | --- | --- |
| no | 248 000 | 152 000 | 158.0 MB → 66.98 MB | **yes** (cap 67.11 MB) |
| yes | 174 000 | 226 000 | 203.6 MB → 167.7 MB | **no — stopped 2.5x over cap** |
Two consequences, both real:
1. **A node whose peer is down cannot enforce its event-log cap at all.** Nothing drains the
oplog, so the file never shrinks and the guard stops the trim every tick, forever.
2. **The "1 GB event log" budget was in practice shared with replication bookkeeping the event
log does not own.** Worse, the direction of the coupling is backwards: a *replication fault*
(peer down, oplog piling up) would, under a naive "just keep deleting" fix, cause operational
event history to be destroyed. The cap must not be a function of the peer's health.
The old guard was not the bug — it was doing its job, preventing the table being wiped in pursuit
of an unreachable target. The bug is that the target was unreachable, because the metric was
wrong.
## 2. `dbstat` is not available — the decision this memo turns on
The register proposed `SELECT sum(pgsize) FROM dbstat WHERE name='site_events'`. `dbstat` is a
**compile-time option** (`SQLITE_ENABLE_DBSTAT_VTAB`), and it was verified before any
implementation was written. It is **not enabled** in the native library this solution ships
(`SQLitePCLRaw.lib.e_sqlite3`, the security-pinned 2.1.12 build referenced by both
`ZB.MOM.WW.ScadaBridge.SiteEventLogging` and its test project).
Probed live against a real file, SQLite **3.53.3**:
```
PRAGMA compile_options -> no ENABLE_DBSTAT_VTAB
(ENABLE_FTS3/4/5, RTREE, GEOPOLY, SESSION, PREUPDATE_HOOK, … present)
pragma_module_list -> 10 modules: fts3, fts3tokenize, fts4, fts4aux, fts5, fts5vocab,
geopoly, pragma_module_list, rtree, rtree_i32
SELECT sum(pgsize) FROM dbstat -> SQLite Error 1: 'no such table: dbstat'
SELECT … FROM dbstat('main') WHERE name=… -> SQLite Error 1: 'no such table: dbstat'
```
`sqlite_dbpage` — the other route to per-table page attribution — is absent for the same reason.
The module list is non-empty, so this is a genuine absence and not an empty-result artefact.
**This is pinned by a committed test**, `SqliteCapabilityTests.DbStat_IsNotAvailable_…`, written to
fail if `dbstat` ever appears. If it does, the right move is to delete the model in §3 and measure
exactly; the test's own comment says so.
Alternatives considered and rejected:
- **Switch to a SQLite build that enables `dbstat`** (system `libsqlite3` via
`SQLitePCLRaw.bundle_sqlite3`, or a vendored native build). Rejected: the provider is shared with
`ZB.MOM.WW.LocalDb` and every other SQLite consumer in the product, so this trades a storage-cap
accuracy problem for fleet-wide version skew across macOS dev boxes and the Debian-based runtime
image — a disproportionate blast radius for a coarse safety valve.
- **Give `site_events` its own file again.** Reverses LocalDb Phase 1/2 consolidation and removes
the table from CDC replication. Out of the question.
- **Maintain a running byte counter in the writer, or via triggers.** The writer is not the only
thing that inserts into `site_events` (replication applies remote rows), a counter does not
survive a restart, and adding our own triggers to a `RegisterReplicated` table is hazardous —
LocalDb builds its capture triggers from the column list at registration time and extra triggers
would also fire on replicated applies.
- **Keep the whole-file metric and simply stop re-measuring mid-loop** (compute a delete plan once
and execute it). This makes the trim terminate, but it makes the *event log* pay for bookkeeping
bytes: an oplog inflated by a down peer would drive aggressive deletion of operational history.
That is consequence 2 above, made worse rather than fixed.
## 3. What the cap measures now
`EventLogPurgeService.EstimateSiteEventsSize()` returns the live row count and modelled bytes per
row. `MaxStorageMb` is compared against `rowCount × bytesPerRow`.
Row width is **sampled** — `SizeSampleRows` (500) rows from each end of the table, head and tail.
The head is what the oldest-first trim will actually delete; the tail catches an event stream whose
shape has drifted. Widths are taken with `octet_length()` (SQLite 3.43+), not `length()`, which
counts characters and would under-size any multi-byte UTF-8 message; a second committed test pins
that function's availability.
The table term is computed as **whole pages**, `ceil(rows / rowsPerPage) × pageSize`, rather than
as a flat bytes-per-row rate:
```
usablePerPage = (pageSize - 35) * 0.78 // max-local-payload rule × measured mean page fill
avgCellBytes = avgPayloadBytes + 16 // payload-length varint, rowid varint, cell pointer
rowsPerPage = max(1, floor(usablePerPage / avgCellBytes))
tableBytes = ceil(rowCount / rowsPerPage) * pageSize
indexBytes = rowCount * (avgIndexedBytes + 8 * 4) / 0.75
estimate = min(tableBytes + indexBytes, wholeFileLogicalBytes)
```
Whole pages rather than a rate because that term **dominates once rows approach half a page**: a
2 KB row cannot be packed two-to-a-page, so each one wastes the remainder. A linear model
underestimates that case by ~40 % (measured), and underestimating is the failure direction that
matters for a cap.
**Index pages count against the cap — deliberately.** Measured, the four indexes are ~34 % of
`site_events`' total footprint (397.2 vs 260.9 bytes/row with and without them, same 200 000 rows),
and the 1 GB default was sized against the 393 bytes/row figure from row 34, which already included
them. Excluding index pages would silently raise real disk consumption by about a third at an
unchanged setting. The index term is driven by the *sampled* widths of the four indexed columns, so
an event stream that leaves `instance_id` null is charged less than one that fills it.
**Clamped by the file.** The whole-file logical size is a true upper bound on any single table's
pages, so the model is capped by it. This can only ever correct an over-estimate downward, and it
cannot reintroduce the row-81 defect: on a replicated node the file is *larger* than
`site_events`, never smaller, so the clamp does not bind there.
`GetDatabaseSizeBytes()` survives, renamed in meaning rather than in signature: it is the whole
consolidated file's logical size, used for the clamp and as a diagnostic. It is no longer what the
cap compares against.
### 3.1 Measured accuracy
Calibration seeded realistic rows into a real file and measured `page_count`/`freelist_count`/
`page_size`. Two independent cross-checks against row 34's harness: the baseline workload came out
at **397.2 bytes/row** against row 34's independently measured **393**, and the row-count-at-1 GB
figure follows.
| Workload | rows | actual bytes/row | payload bytes/row |
| --- | ---: | ---: | ---: |
| 1-in-10 details @200 B (baseline) | 200 000 | 397.2 | 187.9 |
| same, **no indexes** | 200 000 | 260.9 | 187.9 |
| no details at all | 200 000 | 373.5 | 167.7 |
| details on every row @200 B | 200 000 | 597.7 | 367.7 |
| details on every row @2 KB | 50 000 | 4295.2 | 2215.7 |
| 1-in-10 details @4 KB | 50 000 | 824.7 | 583.0 |
| small table | 5 000 | 396.5 | 189.6 |
The shipped model, scored against a **differential** measurement of the table's real page cost
(file measured with ballast only, events added, growth attributed to `site_events`) — this is
`EstimateSiteEventsSize_TracksTheRealFootprint`, run on every build:
| Workload | estimated | actual | error |
| --- | ---: | ---: | ---: |
| narrow rows, no details | 3 155 626 | 2 912 256 | **+8.4 %** |
| typical rows, 200 B details | 3 992 096 | 3 538 944 | **+12.8 %** |
| wide rows, 2 KB details | 12 748 048 | 12 845 056 | **−0.8 %** |
The test asserts the band **−5 % … +25 %**. The bias is intentionally to the high side: a cap that
fires slightly early forfeits a slice of the budget, whereas one that reads low lets the log
overrun the disk allowance the cap exists to defend. Against the previous behaviour's **+250 %**
error on a replicated node this is not a close call.
**Known limitation.** The model samples 1000 rows out of a table that may hold millions. An event
stream whose row width changes sharply *in the middle* of the history — neither at the head nor the
tail — is mis-sized until the trim reaches it. Self-correcting in practice, since the cap is
re-evaluated every tick and the head sample advances as the oldest rows go.
## 4. The wipe-prevention guard, restated
The register asked whether the "size did not decrease" guard should survive. It should, and it
does, but its meaning has changed and it has gained a sibling. There are now **two** backstops:
1. **Unusable row width → refuse to trim.** Checked *before* the cap comparison, because a zero
bytes-per-row makes "is `site_events` over cap" unanswerable rather than false. Logs a Warning
naming the row count and deletes nothing. The old code had no analogue: its answer to an
unanswerable cap was to keep deleting until the file shrank, which is exactly how a cap that can
never be met empties a table.
2. **Estimated size did not decrease across a batch → stop.** The old guard, re-based onto the
table-scoped metric. Crucially it **can no longer be tripped by replication bookkeeping** — the
metric excludes it — so when it fires it now means precisely one thing: the table is not
shrinking even though deletes are landing, i.e. **new events are arriving at least as fast as
the trim removes them.** That is a real operational condition and the Warning says so.
For this to be observable rather than tautological, the loop re-reads the **row count** from the
database after each batch instead of decrementing a local counter, so concurrent recorder writes
are visible. Bytes-per-row, by contrast, is sampled **once per tick and held fixed** — re-sampling
every batch would make the estimate jitter as the trim consumes the head of the table, and a
sample that happened to widen could stall the loop through this very guard. That is the same
class of self-defeating measurement as row 81 itself, and it is deliberately avoided.
Both backstops are covered by tests that pin a degenerate estimate through `internal virtual`
seams (`EstimateSiteEventsSize`, `GetSiteEventsRowCount`).
## 5. Behaviour on an UNREPLICATED node
Materially unchanged, as intended — but not bit-identical, and the delta is worth stating.
On a node with no CDC triggers there is no oplog and no tombstone table, so whole-file logical size
was already ≈ `site_events` + the nine other (largely small) tables. The cap therefore used to fire
at roughly the right place, and the row-34 measurement confirms it: the unreplicated trim reached
66.98 MB against a 67.11 MB cap.
What changes:
- **Slightly more headroom.** The other nine tables no longer count against the event log's budget.
On a rig-shaped site those tables are small relative to a 1 GB event log, so the shift is a few
percent at most.
- **Model error replaces measurement error.** Where the old number was exact-but-wrong-scoped, the
new one is right-scoped but carries the +8 %/+13 % conservative bias of §3.1. Net effect on an
unreplicated node: the cap fires marginally earlier than before at the same setting.
Neither warrants a config change. Sites that were happy with `MaxStorageMb: 1024` remain so.
## 6. Explicitly out of scope: LocalDb's own bookkeeping
This change makes the cap ignore `__localdb_oplog`, `__localdb_row_version` and
`__localdb_snapshot_state`. It does **not** attempt to manage them, and should not.
Their size is governed by LocalDb's own machinery — `LocalDb:Replication:MaxOplogRows` (default
1 000 000) and `TombstoneRetention` (default 7 days) — and row 34's pacing
(`MaxPurgedRowsPerTick`) is what keeps a purge burst from blowing through the oplog cap and forcing
a snapshot resync. Those are the correct controls and they already exist. A second, competing
bound applied from the event log would only obscure which subsystem is responsible for a full disk.
The honest statement of the budget is therefore: **`MaxStorageMb` bounds `site_events` and its
indexes; LocalDb replication bookkeeping is separate overhead on the same volume, bounded
separately.** Sizing a site's data volume means adding the two, and the topology guide now says so.
Row 34's measurement gives the scale — a 2.73 M-row purge left 707 MB of bookkeeping in the file
after deleting a 1 GB event log — and that overhead is transient, draining as the peer acks and as
tombstones expire.
## 7. Verification
- **`dbstat` unavailability probed before any implementation was written**, on a real file, with
the result recorded in §2 and pinned by `SqliteCapabilityTests`.
- **Fail-before, row 81.** With `EstimateSiteEventsSize` reverted to pre-row-81 semantics (whole
file attributed to `site_events`) and nothing else changed, **5 of the new tests fail** and all
99 others pass. `PurgeByStorageCap_WithLargeBallastInTheSameFile_StillEnforces…` reproduces the
original signature exactly: `remaining=2000 estimate=21725184 cap=11534336` — 4000 of 6000 rows
destroyed and the cap **still unmet**.
- **Fail-before, row 84.** With per-phase emission restored,
`RunPurge_WhenBothPhasesDeleteRows_LogsTheBacklogNoteOnceForTheWholeTick` fails with
`Assert.Single() Failure: The collection contained 2 matching items`.
- `dotnet build ZB.MOM.WW.ScadaBridge.slnx` — clean, 0 warnings.
- `ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests`: **104 passed / 0 failed** (93 before this change).
- The docker rig was not touched.
## 8. Register row 84, folded in
`LogReplicationBacklogNote` was emitted by each purge phase, so a tick in which the retention purge
*and* the cap trim both deleted rows produced two identical Information notes. It is now emitted
once from `RunPurge`, driven by the tick's shared `PurgeBudget`.
The budget is the natural source: it already holds exactly "rows this tick deleted", which is also
the more useful number to put next to an oplog spike — the tick's whole contribution rather than
one phase's share. `PurgeByRetention` keeps its own "Purged N events older than D days" line; only
the replication note moved.
docs/plans/sql/AddBundleUnlockAttemptTable.sql
+45
View File
@@ -0,0 +1,45 @@
-- Register row 52: the fleet-wide bundle-unlock throttle window.
--
-- Creates the central [BundleUnlockAttempt] table that both central nodes share, so
-- ScadaBridge:Transport:MaxUnlockAttemptsPerIpPerHour is a fleet cap rather than a
-- per-node, per-process one. Operational history, not audit: non-partitioned, standard
-- [PRIMARY] filegroup, no DB-role restriction, and rows are pruned opportunistically by
-- the application once they age past twice the trailing window.
--
-- Run with QUOTED_IDENTIFIER ON. Several tables in this database carry filtered indexes
-- and persisted computed columns, and SQL Server refuses DDL/DML against them with error
-- 1934 when the session has QUOTED_IDENTIFIER OFF -- which is exactly what `sqlcmd` gives
-- you by default (pass -I, or keep this header). The SETs sit in their own batch so they
-- are in force when the statements after the GO parse.
SET QUOTED_IDENTIFIER ON;
SET ANSI_NULLS ON;
GO
BEGIN TRANSACTION;
IF NOT EXISTS (
SELECT * FROM [__EFMigrationsHistory]
WHERE [MigrationId] = N'20260815154803_AddBundleUnlockAttemptTable'
)
BEGIN
EXEC(N'CREATE TABLE [BundleUnlockAttempt] (
[Id] bigint NOT NULL IDENTITY,
[ClientKey] varchar(64) NOT NULL,
[AttemptedAtUtc] datetime2 NOT NULL,
CONSTRAINT [PK_BundleUnlockAttempt] PRIMARY KEY ([Id])
)');
EXEC(N'CREATE INDEX [IX_BundleUnlockAttempt_Client_Attempted]
ON [BundleUnlockAttempt] ([ClientKey], [AttemptedAtUtc])');
END;
IF NOT EXISTS (
SELECT * FROM [__EFMigrationsHistory]
WHERE [MigrationId] = N'20260815154803_AddBundleUnlockAttemptTable'
)
BEGIN
INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion])
VALUES (N'20260815154803_AddBundleUnlockAttemptTable', N'10.0.7');
END;
COMMIT;
GO
docs/plans/sql/AddDeploymentRecordListSortIndex.sql
+56
View File
@@ -0,0 +1,56 @@
-- Register row 35: the deployment-list sort key gets its own index.
--
-- Replaces the single-column [IX_DeploymentRecords_DeployedAt] with the composite
-- [IX_DeploymentRecords_DeployedAt_Id] ([DeployedAt] DESC, [Id] DESC), which is exactly
-- the order DeploymentManagerRepository.QueryDeploymentListPageAsync (and
-- QueryDeploymentSummariesAsync) asks for. With both key columns descending, a forward
-- scan delivers the requested order, so a paged Skip/Take reads a Top-N off the index
-- instead of sorting the whole filtered set on every page request.
--
-- The DROP is safe rather than merely tolerable. [DeployedAt] leads both indexes, so any
-- predicate the old one served the composite serves too (direction does not affect
-- seekability -- SQL Server can scan either way), and the leaf rows are the same width:
-- [Id] is the clustered primary key and was therefore already in the old nonclustered
-- index's leaf as the row locator. All that changes is that [Id] becomes a declared,
-- ordered key column. Keeping both would maintain a second index on every deployment
-- insert and status transition for no read benefit.
--
-- Note [SystemArtifactDeploymentRecords] is a DIFFERENT table with its own
-- [DeployedAt] index; it is deliberately untouched.
--
-- DeploymentRecords is small enough that a plain (offline) index rebuild is fine. On an
-- Enterprise edition instance where the deployment history is large and the Central UI
-- must stay responsive, add WITH (ONLINE = ON) to the CREATE INDEX below.
--
-- Run with QUOTED_IDENTIFIER ON. Several tables in this database carry filtered indexes
-- and persisted computed columns, and SQL Server refuses DDL/DML against them with error
-- 1934 when the session has QUOTED_IDENTIFIER OFF -- which is exactly what `sqlcmd` gives
-- you by default (pass -I, or keep this header). The SETs sit in their own batch so they
-- are in force when the statements after the GO parse.
SET QUOTED_IDENTIFIER ON;
SET ANSI_NULLS ON;
GO
BEGIN TRANSACTION;
IF NOT EXISTS (
SELECT * FROM [__EFMigrationsHistory]
WHERE [MigrationId] = N'20260815165452_AddDeploymentRecordListSortIndex'
)
BEGIN
EXEC(N'DROP INDEX [IX_DeploymentRecords_DeployedAt] ON [DeploymentRecords]');
EXEC(N'CREATE INDEX [IX_DeploymentRecords_DeployedAt_Id]
ON [DeploymentRecords] ([DeployedAt] DESC, [Id] DESC)');
END;
IF NOT EXISTS (
SELECT * FROM [__EFMigrationsHistory]
WHERE [MigrationId] = N'20260815165452_AddDeploymentRecordListSortIndex'
)
BEGIN
INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion])
VALUES (N'20260815165452_AddDeploymentRecordListSortIndex', N'10.0.7');
END;
COMMIT;
GO
docs/requirements/Component-CLI.md
+20 -3
View File
@@ -194,8 +194,8 @@ scadabridge data-connection certs remove --site <identifier> --thumbprint <hex>
```
scadabridge external-system list
scadabridge external-system get --id <id>
scadabridge external-system create --name <name> --endpoint-url <url> --auth-type <type> [--auth-config <json>]
scadabridge external-system update --id <id> [--name <name>] [--endpoint-url <url>] [--auth-type <type>] [--auth-config <json>]
scadabridge external-system create --name <name> --endpoint-url <url> --auth-type <type> [--auth-config <json>] [--timeout-seconds <n>] [--max-retries <n>] [--retry-delay-seconds <n>]
scadabridge external-system update --id <id> [--name <name>] [--endpoint-url <url>] [--auth-type <type>] [--auth-config <json>] [--timeout-seconds <n>] [--max-retries <n>] [--retry-delay-seconds <n>]
scadabridge external-system delete --id <id>
scadabridge external-system method list --external-system-id <id>
scadabridge external-system method get --id <id>
@@ -238,9 +238,26 @@ record of admin edits). `audit-log` is a deprecated alias that still works but e
warning; migrate scripts to `audit-config`.
```
scadabridge audit-config query [--user <username>] [--entity-type <type>] [--action <action>] [--from <date>] [--to <date>] [--page <n>] [--page-size <n>]
scadabridge audit-config query [--user <username>] [--entity-type <type>] [--action <action>] [--from <date>] [--to <date>] [--page <n>] [--page-size <n>] [--detail]
```
The response is an `AuditLogQueryResult` — `{ "entries": [ … ], "totalCount": <n> }`, where
`totalCount` is the unpaged match count. The result type is a named record specifically
because the handler must not return the repository's `(Entries, TotalCount)` ValueTuple:
`System.Text.Json` serializes properties only, so a tuple result renders as the literal
`{}` and every query looks empty regardless of how many rows matched. `ManagementActor.SerializeResult`
now rejects tuple results outright so the failure mode cannot recur silently.
JSON is the primary format here and is never altered by the table path. `--format table`
applies a compact projection (`AuditConfigTableProjection`) that unwraps the envelope into
one row per entry — `id`, `timestamp`, `user`, `action`, `entityType`, `entityName`,
`entityId` — and drops `afterStateJson`, the full post-change entity snapshot, since a
single one fills the terminal. Without a projector the renderer took its object branch and
printed the whole `entries` array as raw JSON inside one `Property`/`Value` cell (register
row 57). `totalCount` survives the unwrap as a trailing `note` row, emitted only when the
page is a strict subset of the matches. `--detail` skips the projection and renders the
raw payload, mirroring `template list`/`template get`.
The new centralized Audit Log component (#23) is exposed via the `scadabridge audit` group below.
### Centralized Audit Commands
docs/requirements/Component-CentralUI.md
+8 -1
View File
@@ -13,6 +13,12 @@ Central cluster only. Sites have no user interface.
- **Framework**: Blazor Server (ASP.NET Core). UI logic executes on the server, updates pushed to the browser via SignalR.
- Keeps the entire stack in C#/.NET, consistent with the rest of the system (Akka.NET, EF Core).
- SignalR provides built-in support for real-time UI updates.
- **Dialog service confirm labels.** Every confirmation goes through `IDialogService.ConfirmAsync(title, message, danger, confirmLabel)`, rendered by the single `DialogHost` in `MainLayout`. `danger` selects **styling only** (`btn-danger`); `confirmLabel` names the action. When no label is supplied the button falls back to "Delete" for a danger confirm and "Confirm" otherwise — a convenience for the ~20 genuine delete call sites, and the reason the Health dashboard's *Trigger failover* dialog used to offer a red **Delete** button (deferred-work register row 28). **Any destructive action that is not literally a delete must pass `confirmLabel`**: failover → *Fail over*, secured-write approve/reject → *Approve* / *Reject*, instance disable → *Disable*, and every parked-message / notification / cached-call discard → *Discard*.
- **Dialog confirm severity — three rungs, not two** (deferred-work register row 75). `DialogSeverity` drives the confirm button's colour: `Normal` → `btn-primary`, `Caution` → `btn-warning`, `Destructive` → `btn-danger`. Only `Destructive` inherits the "Delete" label fallback above; an unlabelled `Caution` reads *Confirm*, because a caution action is by definition not a delete.
- **Why a middle rung exists.** The boolean was a two-state flag for a three-state concept, so a consequential-but-not-destructive action had to choose between looking like a delete and looking like the safe default. The **Secured Writes** page is where that bit: *Approve* and *Reject* both rendered as identical red buttons, so at the moment of a two-person-control decision the colour carried **no discriminating information at all** — and the affirmative half of the control wore the delete affordance.
- **The decision.** *Approve* is `Caution` (amber): approving executes a real write to live plant equipment, so it is not a benign primary action, but it is not destroying anything either. *Reject* **keeps** `Destructive` (red): it is terminal and irreversible for the request — the Operator must resubmit — and holding it red is what keeps the two opposite decisions visually distinct. Dropping *Reject* to a neutral button was considered and rejected: it would simply invert the bias, making declining look like the safe default.
- Colour is never the only signal — the button always carries its verb (see the label contract above), so the mapping is an accessibility affordance, not the meaning.
- **`DialogSeverity` is an additive overload**, not a replacement: the boolean form remains the signature for the ~34 existing call sites, which are genuinely two-state. Adding a fifth optional parameter to the boolean form instead would have silently un-matched every existing four-argument NSubstitute stub — the row-28 gotcha, where a stub that stops matching returns a null `Task` and the caller NREs with no compile error. Reach for the overload only when `Caution` is what you mean.
## Failover Behavior
@@ -225,7 +231,7 @@ Per-leaf alarm rendering (leaf nodes are individual conditions for native alarms
- A **Secured Writes** page (`/operations/secured-writes`) drives the **two-person** authorization workflow for writes through the MxAccess Gateway: an **Operator** initiates the write, a separate **Verifier** approves it, and only an approved write reaches the site.
- The **page itself is gated by `RequireSecuredWriteAccess`** — satisfied by any of `Operator` / `Verifier` / `Administrator`. Although the pending/history lists are read-only, they expose process-sensitive tag values, so the page is not open to every authenticated user; this aligns with the ManagementActor `ListSecuredWritesCommand` any-of gate (arch-review UA1). The submit and approve/reject sub-actions are further gated by `RequireOperator` / `RequireVerifier` respectively (below).
- **Operator (submit)** — a submit form gated by `RequireOperator`: pick the site → an **MxGateway** connection on that site → the tag path → a typed value → an optional comment. Submission inserts a `Pending` `PendingSecuredWrite` row centrally; it does **not** write anything yet.
- **Verifier (approve / reject)** — a pending queue gated by `RequireVerifier` with **Approve** / **Reject** (+comment) actions. Approve shows a confirmation of the exact site / connection / tag / value before firing, and surfaces the **submission age** (`Submitted <age> ago (<UTC timestamp>)`) so a verifier can catch a stale setpoint before it reaches the device. The verifier's **own submissions are disabled in the UI and rejected server-side** (no self-approval). On approve, central marks the row `Approved` and relays the write to the site MxGateway (records `Executed` / `Failed`); reject moves it to `Rejected` with a reason. The pending queue is **unpaged** — it is bounded by the submission **TTL**: a `Pending` row that is neither approved nor rejected before the TTL elapses expires to the terminal **`Expired`** status and drops out of the queue into History.
- **Verifier (approve / reject)** — a pending queue gated by `RequireVerifier` with **Approve** / **Reject** (+comment) actions. Approve shows a confirmation of the exact site / connection / tag / value before firing, and surfaces the **submission age** (`Submitted <age> ago (<UTC timestamp>)`) so a verifier can catch a stale setpoint before it reaches the device. The two confirmations are deliberately styled apart: **Approve is amber (`DialogSeverity.Caution`) and Reject stays red** — see the confirm-severity contract under Technology for why (register row 75). The verifier's **own submissions are disabled in the UI and rejected server-side** (no self-approval). On approve, central marks the row `Approved` and relays the write to the site MxGateway (records `Executed` / `Failed`); reject moves it to `Rejected` with a reason. The pending queue is **unpaged** — it is bounded by the submission **TTL**: a `Pending` row that is neither approved nor rejected before the TTL elapses expires to the terminal **`Expired`** status and drops out of the queue into History.
- **History** — terminal rows (Executed / Failed / Rejected / **Expired**, where `Expired` is a real TTL-driven terminal status) with the full who/when/outcome trail (operator, verifier, comments, timestamps, any execution error). Terminal rows accumulate without bound, so the History table is **offset-paged** (page-size 50, Prev/Next) driven by the server's unpaged `TotalCount` (`ListSecuredWritesCommand` `Skip`/`Take`); the pager buttons disable at the real bounds.
- Every lifecycle event (submit / approve / reject / execute) is written to the central Audit Log; the rows share the `PendingSecuredWrite.Id` as `CorrelationId` so they join into one operation (see Component-ManagementService.md, Component-AuditLog.md).
- **Dev caveat**: with `DisableLogin` on, the auto-login identity holds all roles, so the two-person flow cannot be exercised end-to-end by a single user via the dev UI — no-self-approval is covered by handler tests; real two-person use requires two real identities.
@@ -243,6 +249,7 @@ Per-leaf alarm rendering (leaf nodes are individual conditions for native alarms
- Headline **Notification Outbox KPI tiles** — queue depth, stuck count, and parked count. These are central-computed by the Notification Outbox from the central `Notifications` table (not part of any site health report). The full outbox view is on the dedicated Notification Outbox page.
- Headline **Site Call Audit KPI tiles** — buffered count, parked count, and failed-last-interval. These are central-computed by the Site Call Audit component from the central `SiteCalls` table (not part of any site health report). The full cached-call view is on the dedicated Site Calls page.
- Headline **Audit KPI tiles** — three tiles in a new "Audit" KPI group: **Audit volume**, **Audit error rate**, and **Audit backlog**. These are sourced from the Audit Log component (#23) and Health Monitoring per the metric definitions in Component-HealthMonitoring.md; the dashboard simply surfaces them. The full audit query view is on the dedicated Audit Log page.
- **Trigger failover** (`CentralFailoverControl`, Administrator only) — on the central card and on each site card. Disabled with an explanatory tooltip when there is no standby to take over. It always confirms first, and the confirmation's red button reads **Fail over** (see the dialog-label contract under Technology); the central variant's copy states that this page is served by the node being restarted and will disconnect and reconnect, while the site variant deliberately does not, since a site is a different cluster.
- **Shared KPI cache (arch-review WP2.4).** All four KPI families above — Notification Outbox, Site Call Audit, Audit, and their per-site/per-node breakdowns — are read through a process-level `IKpiSnapshotCache`, not queried per page load. Each accessor is independently memoized (8s TTL) with single-flight production, so N Blazor circuits polling the same KPI inside one window (e.g. ten operators with the Health dashboard open) cost the node ONE aggregate SQL round trip per KPI per window, not N. A failed query is never memoized — the next caller re-attempts it, and the calling page's existing per-tile "unavailable" degradation is unchanged. Every KPI-tile page (Health, Notification Outbox, Site Calls, Audit Log) shares the same cache instance.
### Site Event Log Viewer (Deployment Role)
docs/requirements/Component-Communication.md
+1 -1
View File
@@ -108,7 +108,7 @@ The streaming protocol is defined in `sitestream.proto` (`src/ZB.MOM.WW.ScadaBri
- **Service**: `SiteStreamService` — hosted on each site node by `SiteStreamGrpcServer` — exposes six RPCs. Two are real-time **server-streaming** subscriptions (`SubscribeInstance`, `SubscribeSite`); the other four are **unary request/response** calls added by the Audit Log (#23) and Site Call Audit (#22) components. These are distinct from the command/control gRPC services (`CentralControlService` on central, `SiteCommandService` on the site) — `SiteStreamService` is the streaming + audit-pull surface, not the command/control channel:
- `SubscribeInstance(InstanceStreamRequest) returns (stream SiteStreamEvent)` — the per-instance real-time debug stream (§6).
- `SubscribeSite(SiteStreamRequest) returns (stream SiteStreamEvent)` — the **site-wide, alarm-only** aggregated stream (§6.1) added for the operator Alarm Summary live cache. Additive (new RPC + new `SiteStreamRequest { correlation_id }` message; no field renumbering). The server handler mirrors `SubscribeInstance` — but with **its own, larger send channel** (`GrpcSiteAlarmStreamChannelCapacity`, default 20 000, `DropOldest`; WP2.3): sharing the Debug View's 1000-slot channel meant an alarm burst during a WAN stall silently evicted operator-visible transitions to make room for diagnostics traffic. Drops are tagged `stream=site-alarms` on `scadabridge.site.stream.events_dropped`. Otherwise identical — same `GrpcMaxConcurrentStreams`/`GrpcMaxStreamLifetime` limits, `SiteConnectionOpened/Closed` telemetry, and `StreamRelayActor` mapping — but subscribes via `SiteStreamManager.SubscribeSiteAlarms` (all instances, `AlarmStateChanged` only; attribute events dropped, no per-instance filter).
- `SubscribeSite(SiteStreamRequest) returns (stream SiteStreamEvent)` — the **site-wide, alarm-only** aggregated stream (§6.1) added for the operator Alarm Summary live cache. Additive (new RPC + new `SiteStreamRequest { correlation_id }` message; no field renumbering). The server handler mirrors `SubscribeInstance` — but with **its own, larger send channel** (`GrpcSiteAlarmStreamChannelCapacity`, default 20 000, `DropOldest`; WP2.3): sharing the Debug View's 1000-slot channel meant an alarm burst during a WAN stall silently evicted operator-visible transitions to make room for diagnostics traffic. Drops are tagged `stream=site-alarms` on `scadabridge.site.stream.events_dropped`. Otherwise identical — same `GrpcMaxConcurrentStreams`/`GrpcMaxStreamLifetime` limits, `SiteStreamSubscriberOpened/Closed` telemetry, and `StreamRelayActor` mapping — but subscribes via `SiteStreamManager.SubscribeSiteAlarms` (all instances, `AlarmStateChanged` only; attribute events dropped, no per-instance filter).
- `IngestAuditEvents(AuditEventBatch) returns (IngestAck)` — a legacy central-side ingest surface on the *site*-hosted service; it is **dead in the shipped topology** because no site ever dials a site for ingest. The production audit-telemetry *push* path is site→central gRPC to `CentralControlService`, which routes the batch to the central `AuditLogIngestActor` proxy and returns the accepted `EventId`s.
- `IngestCachedTelemetry(CachedTelemetryBatch) returns (IngestAck)` — ingest receiving surface for the combined cached-call telemetry packet (audit row + `SiteCalls` operational upsert written in one transaction).
- `PullAuditEvents(PullAuditEventsRequest) returns (PullAuditEventsResponse)` — central→site **reconciliation pull** for the Audit Log self-heal feed; the site serves `Pending`/`Forwarded` rows from its `ISiteAuditQueue`. **At-least-once (WP2.3):** rows are NOT retired when served. The site retires (`Reconciled`) everything at or before the cursor of the **next** pull, because that cursor is central's only proof of receipt — a fault between the response leaving the site and central committing it re-serves the rows instead of losing them (central dedups on `EventId`). The request carries an additive composite-keyset `after_id` (field 3) mirroring `PullSiteCallsRequest`; with it the read is a strict `(OccurredAtUtc, EventId)` keyset and the retirement is exact, without it the legacy inclusive `>=` read applies and only rows strictly older than the cursor instant are provably received.
docs/requirements/Component-ConfigurationDatabase.md
+4 -1
View File
@@ -120,9 +120,12 @@ The configuration database stores all central system data, organized by domain a
- **Site Scoping Rules**: Per-mapping site scope restrictions for Deployment role.
### Deployment
- **Deployment Records**: Deployment history per instance (timestamp, user, status, deployed configuration snapshot).
- **Deployment Records**: Deployment history per instance (timestamp, user, status, deployed configuration snapshot). Indexes: unique on `DeploymentId` (deployment identity / idempotency), `InstanceId` (FK-backing, and the per-instance history read), and `IX_DeploymentRecords_DeployedAt_Id (DeployedAt DESC, Id DESC)` — the deployment list's exact sort key, so a paged read takes a Top-N off the index instead of sorting the whole filtered set per page. `Id` is in the key as the deterministic tie-break: a bulk site deploy stamps many rows in the same `DeployedAt` instant, and without it a row can repeat or vanish across page boundaries. This index **replaced** an earlier single-column one on `DeployedAt` (migration `AddDeploymentRecordListSortIndex`, deferred-work register row 35) rather than joining it — `DeployedAt` leads both, and `Id` is the clustered PK so it was already in the old index's leaf as the row locator, making the old index strictly redundant at identical leaf width.
- **System-Wide Artifact Deployment Records**: Deployment history for shared artifacts (timestamp, user, artifact type, status).
### Transport
- **Bundle Unlock Attempts**: One row per permitted bundle-unlock passphrase attempt (`ClientKey`, `AttemptedAtUtc`), forming the fleet-wide sliding window behind Transport's `MaxUnlockAttemptsPerIpPerHour` throttle — see `Component-Transport.md` (register row 52). Operational history, not audit: non-partitioned, no DB-role restriction, and rows are pruned opportunistically by the application once they age past twice the window. **Denied** attempts are deliberately not recorded (a denial burns no budget), and no bundle identity is stored — the durable record of a failed unlock is the `BundleImportUnlockFailed` audit row. Central-only: written only by the Transport component, which the host registers for central roles.
### Audit Logging
- **Audit Log Entries**: Append-only audit trail (timestamp, user, action, entity type, entity ID, entity name, state as JSON). Stores only the after-state — change history is reconstructed by comparing consecutive entries. Entries are never modified or deleted. No retention policy — retained indefinitely. Indexed on timestamp, user, entity type, entity ID, and action for efficient filtering.
docs/requirements/Component-DataConnectionLayer.md
+7 -1
View File
@@ -355,6 +355,10 @@ Each value update delivered to an Instance Actor includes:
## Connection Actor Model
> **Where connection actors come from — the Site Runtime is the only creator (register row 85)**: the DCL manager creates and removes connection actors **only** when the Deployment Manager tells it to, via `CreateConnectionCommand` / `RemoveConnectionCommand`. It never reads storage; the site's `data_connection_definitions` table has no production reader at all, and the DCL has no independent notion of "the connections this site ought to have". A connection actor's config therefore comes from the `Connections` map inside a deployed instance's **flattened config** (pushed as part of starting that instance) or from the system-wide artifact-deploy path. `HandleCreateConnection` is idempotent on the name — it warns and returns if it already holds that connection — but the Deployment Manager is the side that keeps the traffic quiet, hash-guarding on the last config pushed per name so an unchanged connection sends nothing. Row 85 was the consequence of that ownership split being incompletely honoured on the Site Runtime side: two of the four Instance-Actor start paths omitted the connection push, so instances came up whose every tag subscribe was answered `Unknown connection: <name>` here, forever. The fix is in the Site Runtime (the ensure moved into its single creation funnel) — **not** here; giving the DCL manager its own config source would create a second, divergent authority with no arbiter when the two disagree. See `Component-SiteRuntime.md` and `docs/plans/2026-08-15-instance-sweep-self-heal-design.md` §7.
>
> Two known residual gaps of the same family, **not** closed by row 85's fix: `data_connection_definitions` is one of LocalDb's ten replicated tables, so a **connection-only** artifact deploy replicates to the peer node — but only the node that received the deploy pushes it to its DCL, so the peer keeps serving the old endpoint until an instance is redeployed or the node restarts. The removal side is the mirror image: a connection deleted on the peer leaves a live connection actor on this node.
Each data connection is managed by a dedicated connection actor that uses the Akka.NET **Become/Stash** pattern to model its lifecycle as a state machine:
- **Connecting**: The actor attempts to establish the connection. Subscription requests and write commands received during this phase are **stashed** (buffered in the actor's stash).
@@ -414,7 +418,9 @@ Note: Pre-deployment validation at central does **not** verify that tag paths re
The DCL reports the following metrics to the Health Monitoring component via the existing periodic heartbeat:
- **Connection status**: `connected`, `disconnected`, or `reconnecting` per data connection.
- **Tag resolution counts**: Per connection, the number of total subscribed tags vs. successfully resolved tags. This gives operators visibility into misconfigured templates without needing to open the debug view for individual instances. Both numbers are **derived at report time from the connection actor's authoritative per-tag state**, never accumulated in counters: the total is the number of distinct tag paths at least one instance currently subscribes to (the per-tag counted set that also drives value fan-out and the last-subscriber release decision), and the resolved count is the number of tags for which the adapter currently holds a subscription handle. A tag counts toward the total from the moment an instance registers it, whatever its resolution outcome — resolved, awaiting a resolution retry, or failed at connection level — so resolved can never exceed total. Deriving rather than accumulating is deliberate: increment/decrement counters drifted whenever an unsubscribe landed inside a reconnect window (the reconnect clears the per-tag maps the decrements keyed off), leaking a phantom tag into the reported total on every churn cycle.
- **Tag resolution counts**: Per connection, the number of total subscribed tags vs. successfully resolved tags. This gives operators visibility into misconfigured templates without needing to open the debug view for individual instances. Both numbers are **derived from the connection actor's authoritative per-tag state**, never accumulated in counters: the total is the number of distinct tag paths at least one instance currently subscribes to (the per-tag counted set that also drives value fan-out and the last-subscriber release decision), and the resolved count is the number of tags for which the adapter currently holds a subscription handle. A tag counts toward the total from the moment an instance registers it, whatever its resolution outcome — resolved, awaiting a resolution retry, or failed at connection level — so resolved can never exceed total. Deriving rather than accumulating is deliberate: increment/decrement counters drifted whenever an unsubscribe landed inside a reconnect window (the reconnect clears the per-tag maps the decrements keyed off), leaking a phantom tag into the reported total on every churn cycle.
The pair is **pushed to the health collector, not pulled from it**, so it is a cached value that is only as fresh as its last push — and the collector holds it until the next one. Every site that mutates either underlying set must therefore push: entering the Connected state (a legitimate 0/0 — no instance has subscribed yet), the first-subscribe apply, the batch apply (reconnect re-subscribe and resolution probe), and unsubscribe. Missing the first-subscribe push is what made a *perfectly healthy* connection report 0/0 indefinitely — the connect-time 0/0 was never superseded, because a connection whose tags all resolve on the first attempt never reaches the batch path and has its resolution-retry timer cancelled outright for having nothing unresolved. That is the reverse of the usual staleness risk, and it blinded the diagnostic exactly when it should have read clean (deferred-work register row 55; the actor's own `GetHealthReport` reply, which really is derived on demand, was correct throughout).
- **Tag quality counters** are pushed on a genuine quality **transition** only, coalesced onto a `QualityFlushInterval` (1s) single-shot timer. Counter arithmetic still runs per message; only the collector push is deferred, and a value whose quality is unchanged moves no counter at all. Health reports poll at 30s, so the coalescing loses nothing. Three paths flush **synchronously** because their correctness depends on it: the bad-quality push on disconnect, unsubscribe, and the reconnect counter reset.
## Dependencies
docs/requirements/Component-ExternalSystemGateway.md
+1 -1
View File
@@ -42,7 +42,7 @@ Each external system definition includes:
**Legacy colon form (deprecated):** a value that does not start with `{` is parsed with the historical colon-split rules for backward compatibility — API Key as `HeaderName:KeyValue` (or a bare `KeyValue` using the default `X-API-Key` header), Basic Auth as `username:password`. This form cannot represent keys or passwords that contain `:` and is retained only so existing rows keep working; new definitions should use the JSON form. Malformed JSON (a value that starts with `{` but does not parse, or yields no usable `key`/`username`) is **not** an exception — the request is sent without the auth header and a warning is logged (the configuration value itself is never logged).
- **Timeout**: Per-system timeout for all method calls (e.g., 30 seconds). Applies to the HTTP request round-trip. `0` (the default) means "unset" — the gateway's configured `DefaultHttpTimeout` applies instead. Set via the Central UI External System form's "Timeout (seconds)" field; ships to sites in the deployment artifact like the other definition fields.
- **Retry Settings**: Max retry count, fixed time between retries (used by Store-and-Forward Engine for transient failures only).
- **Retry Settings**: Max retry count, fixed time between retries (used by Store-and-Forward Engine for transient failures only). Both are **`0` = unset**, in which case the engine's `DefaultMaxRetries` (50) and `DefaultRetryInterval` (30 s) apply — note this is the inverse of the S&F engine's own `MaxRetries` contract, where a stored `0` means *unbounded* retry; the gateway translates an unset `0` to "no per-system override" rather than passing it through. Authored via the Central UI External System form ("Max Retries" / "Retry Delay (seconds)") or `external-system create|update --max-retries --retry-delay-seconds`, and **deployed to sites in the external-system artifact** alongside the timeout, so the site's cached-call buffering honours the configured policy. (Before the row-27 fix the artifact carried no retry fields and the site table had no columns for them, so a centrally-configured policy was silently ignored and every cached call used the engine default.)
- **Method Definitions**: List of available API methods, each with:
- Method name.
- **HTTP method**: GET, POST, PUT, PATCH, or DELETE.
docs/requirements/Component-HealthMonitoring.md
+126 -4
View File
@@ -34,10 +34,11 @@ Site clusters (metric collection and reporting). Central cluster (aggregation an
| Notification Outbox queue depth | Notification Outbox (central) | Count of `Pending` + `Retrying` notifications — central-computed, not site-reported |
| Notification Outbox stuck count | Notification Outbox (central) | Count of `Pending` / `Retrying` notifications older than the configurable stuck-age threshold — central-computed, not site-reported |
| Notification Outbox parked count | Notification Outbox (central) | Count of `Parked` notifications — central-computed, not site-reported |
| `SiteAuditBacklog` | Audit Log (site) | Count of `Pending` rows in the site-local `AuditLog` plus oldest-pending-age plus on-disk bytes. A configurable threshold drives a Health dashboard warning on the affected site tile. |
| `SiteAuditBacklog` | Audit Log (site) | Count of `Pending` rows in the site-local `AuditLog` plus oldest-pending-age plus on-disk bytes. Surfaced on the Audit KPI tiles (which border-highlight any non-zero backlog) and sampled into KPI history as `auditBacklogPending`. **No configurable per-site threshold badge exists for this metric** — earlier revisions of this document described one, but none was ever implemented and no such option exists. The only configurable sustained-threshold site badge is the LocalDb replication backlog (register row 33, below). |
| `SiteAuditWriteFailures` | Audit Log (site) | Count of failed hot-path audit appends at the site since the last health report. |
| `AuditRedactionFailure` | Audit Log (central) | Count of payload redactor errors (over-redacted payloads, safety-net hit) since the last interval. |
| `LocalDbReplicationConnected` | Consolidated site LocalDb | Whether a peer sync session is currently running on this node. **Nullable — null means "no reading", not "disconnected".** Reported only by a node with replication configured. |
| `LocalDbReplicationConnected` | Consolidated site LocalDb | Whether a peer sync session is currently running on this node. **Nullable — null means "no reading", not "disconnected".** Reported only by a node with replication configured; an unconfigured node reports **null**, never `false` (register row 80 — see *Unconfigured replication reports null* below). |
| `StartupReconcilerActive` | Host (site) | Whether this node spawned its `SiteReconciliationActor` at boot. **Tri-state:** null = not reported (the central self-report, or a pre-row-73 site build), `true` = running, `false` = the node booted with no `IDeploymentConfigFetcher` and therefore has **no startup self-heal**. Latched once at boot by the Host — not polled. Drives the *No self-heal* dashboard badge (register row 73). |
| `LocalDbOplogBacklog` | Consolidated site LocalDb | Unacked oplog depth (Prometheus `localdb_oplog_depth`). **Nullable — null means "no reading", NOT "connected with an empty backlog"** (a failed poll rendered as 0 would report a pair that cannot read its own oplog as perfectly healthy). **Expected transient spikes:** the daily `site_events` retention purge and the storage-cap trim are CDC-captured like any other write, so the backlog jumps by the batch size at purge time and drains as the peer acks it — see `Component-SiteEventLogging.md` → Storage and `docs/deployment/topology-guide.md` → *Reading the replication backlog* for the healthy-versus-faulty signature. |
## Reporting Protocol
@@ -90,7 +91,7 @@ The Audit Log spans both sites (hot-path append + telemetry forward) and central
- **Audit volume** — events/min landing in the central `AuditLog` table, shown global plus per-site sparkline; sourced from the Audit Log component on the active central node.
- **Audit error rate** — percent of central `AuditLog` rows with `Status` other than `Success` / `Delivered` / `Enqueued` over a rolling 5-minute window. This is the operational error rate of audited operations (HTTP 5xx, transient failures, parked deliveries, etc.) — NOT the audit writer's own health. Audit-writer issues surface separately via `AuditRedactionFailure`.
- **Audit backlog** — global aggregate of `SiteAuditBacklog` across reporting sites (count of `Pending` site-local audit rows, oldest pending age, on-disk bytes); click drills into a per-site breakdown. The per-site tile surfaces a warning badge when its `SiteAuditBacklog` crosses the configurable threshold.
- **Audit backlog** — global aggregate of `SiteAuditBacklog` across reporting sites (count of `Pending` site-local audit rows, oldest pending age, on-disk bytes); click drills into a per-site breakdown. The tile border-highlights a non-zero aggregate backlog; there is **no** configurable per-site `SiteAuditBacklog` threshold badge (see the note in Monitored Metrics — earlier revisions described one that was never built).
These tiles are **point-in-time** like the Notification Outbox and Site Call Audit KPI tiles — no time-series store; consistent with Health Monitoring's "current status only" philosophy. The site-scoped `SiteAuditBacklog` / `SiteAuditWriteFailures` metrics arrive in the existing site health report; the central-scoped `AuditRedactionFailure` metric is central-computed alongside the existing central KPIs.
@@ -101,6 +102,126 @@ These tiles are **point-in-time** like the Notification Outbox and Site Call Aud
- Notification Outbox and Site Call Audit KPIs are not stored by Health Monitoring; they are computed point-in-time from the central `Notifications` and `SiteCalls` tables respectively each time the dashboard refreshes — consistent with the current-status-only philosophy.
- Site connectivity history (online/offline transitions) may optionally be logged via the Audit Log or a separate mechanism if needed in the future.
## Sustained LocalDb Replication Backlog Badge
The per-site tile shows a **Replication backlog** warning badge when `LocalDbOplogBacklog` has
stayed above `LocalDbBacklogThreshold` (default **1000**) continuously for
`LocalDbBacklogSustainedFor` (default **15 minutes**). Both are configurable under
`ScadaBridge:HealthMonitoring` and validated at startup.
- **Sustained, not instantaneous — and that is the design, not a refinement.** The daily
`site_events` retention purge and the storage-cap trim are CDC-captured like any other write, so
the backlog jumps by the deleted batch size and drains as the peer acks it. A bare
`backlog > threshold` badge would therefore fire on every replicated site every day. Healthy is
defined by the backlog *returning to ~0*, which only a dwell can express. `LocalDbBacklogSustainedFor`
is validated to be at least one `ReportInterval`: a shorter dwell could be satisfied by a single
sample and would collapse back into the bare threshold.
- **Implementation.** `CentralHealthAggregator` holds `SiteHealthState.LocalDbBacklogAboveSince`,
the instant the backlog first rose above the threshold. Reports at or below the threshold clear
it; reports above it leave the original instant intact (a sustained window, not a sliding one).
- **Null handling is load-bearing.** A report with a `null` backlog leaves the anchor *unchanged*
rather than clearing it — null means "no reading" (unreplicated site, or a failed poll), which is
evidence neither of a drain nor of a backlog. An unreplicated site (rig site-b/site-c) reports
null forever, so it never accumulates an anchor and can never be badged.
- Offline sites are not badged; the Offline badge is the signal that matters there.
## KPI History Sampling of Replication State
`SiteHealthKpiSampleSource` samples `localDbOplogBacklog` and `localDbReplicationConnected` (0/1)
into the KPI history store, and the backlog is charted in the Health dashboard's per-site Site
Health Trends panel as **LocalDb Replication Backlog**. This is the retrospective half of the
signal: a point-in-time tile can never answer "did that spike drain?", which is the only thing
distinguishing an expected purge burst from a stalled peer.
Both metrics are **conditionally emitted** — a site reporting `null` writes no sample rather than a
coerced `0`, unlike the `auditBacklogPending` metric alongside them. A fabricated 0 would render a
pair that cannot read its own oplog, or one deliberately not replicated, as perfectly healthy once
a minute forever. Consequently a site's sample count varies: **12 metrics for an unreplicated site,
14 for a replicated one**, and a gap in these two series truthfully means "not replicated / not
reporting". Note `false` and `null` are different states for the connected flag — a configured link
that is down samples as `0`; no link at all samples as nothing. Both fold as **Gauge** in the hourly
rollup catalog (summing an oplog depth, or a 0/1 flag, per hour would be meaningless).
## Unconfigured Replication Reports Null
The two LocalDb fields are produced by `LocalDbReplicationStatusReporter`, a 30 s site-side poll
over `ISyncStatus`. That poll is **gated on whether the node is configured to replicate at all**,
and an unconfigured node publishes `null` for *both* fields.
The gate is load-bearing because the replication engine is registered **unconditionally** on every
site node and sits inert without a peer — so `ISyncStatus` answers `Connected = false` with a real
`0` backlog on a site that was never meant to replicate. Passing that pair through (the behaviour
before register row 80) had two consequences:
- An operator reading a healthy, deliberately-unreplicated site saw `connected: false` and read a
fault that did not exist.
- Every null branch downstream was dead code on the rig. The conditional KPI emission above and the
sustained-badge anchor exclusion below both exist to handle "no reading", but rig site-b/site-c
never sent one — they emitted a permanently misleading `localDbReplicationConnected = 0` series
instead.
`false` therefore has exactly one meaning on this field: **replication is configured on this node
and the peer session is currently down** — a genuine fault. Beware the specific trap that produced
the bug: configured-ness is `PeerAddress` **OR** `ApiKey` (only the dialling half sets a peer
address; the passive half carries the key alone), so "no `PeerAddress`" does **not** mean
"unreplicated".
The gate value is supplied by the **Host** from `SiteLocalDbSetup.ReplicationIsConfigured`, exactly
as `SiteEventLogReplicationCheck` is supplied to Site Event Logging. Health Monitoring never reads
LocalDb configuration itself, and the PeerAddress-OR-ApiKey rule stays in one place.
## Startup Reconciler Presence
`SiteReconciliationActor` is the per-node startup self-heal: a standby that was DOWN during a
deploy reports its local inventory to central on its next boot, fetches the gap and writes it
(and, since register row 67, restarts Instance Actors the boot-time load missed). The Host skips
it when no `IDeploymentConfigFetcher` is resolvable.
That skip used to be a single boot Warning and nothing else, so a node that had silently lost its
entire self-heal path went on reporting itself perfectly healthy (register row 73). It is now
surfaced three ways:
- **`StartupReconcilerActive` on the health report**, latched at boot by the Host in both branches.
- **A boot-time `Error` log line** (raised from Warning — no configuration makes the fetcher
legitimately absent on a site node) plus a **site event** (`deployment` / `Error`), so the
evidence survives after the boot log has scrolled away.
- **A `No self-heal` warning badge** on the site's Health dashboard tile.
Central's treatment is deliberately **Warning, not failure**. Config still reaches the node through
LocalDb replication while it is up; what it has lost is recovery from a deploy that landed while it
was *down*. That is degraded, not offline. Only the explicit `false` is badged — `null` means "not
reported", and badging it would light up the whole fleet the moment central is upgraded ahead of
the sites. Offline sites are not badged, matching the replication-backlog badge.
## Prometheus Metrics (site node)
Distinct from the health *report* above: these are scraped from the site node's `/metrics`
endpoint (`MetricsPort`, default 8084) rather than sent to central.
| Series | Meaning |
| --- | --- |
| `scadabridge_site_data_connections_connected` | Data connections on this node in the `Connected` state |
| `scadabridge_site_data_connections_total` | Data connections known to this node, in any state |
| `scadabridge_site_stream_subscribers_active` | Open central→site gRPC subscriber streams |
- The two `data_connections` gauges (register row 54) are the **only exported data-connection
signal**; before them the health report's `DataConnectionStatuses` was visible in the UI and
the CLI but never as a metric. `connected < total` is the alert condition — neither number
alone distinguishes "a device is down" from "nothing is configured". They are fed directly by
the Data Connection Layer at each connection state transition, deliberately **not** derived
from the health-report path, so the metric survives a dead report pipeline (the exact failure
the *metrics stale* flag exists to catch).
- Both read **0 on the standby node** — data connections are opened by the deployment path on
the active node only. A standby reporting 0 is healthy; alert on the active node, or on
`max()` across the pair.
- `scadabridge_site_stream_subscribers_active` was renamed from `scadabridge_site_connection_up`
by the same row. It has only ever counted open subscriber streams (Debug View + the site-wide
alarm feed), so it correctly reads 0 on a healthy site nobody is watching — the old name
invited reading it as data-connection health, and did cause exactly that misdiagnosis once.
- All three ride the `ZB.MOM.WW.ScadaBridge` meter, already in the `SiteServiceRegistration.ObservedMeters`
allowlist. `ZbTelemetryOptions.Meters` is an **allowlist**: an unlisted meter exports nothing,
with no error and no missing-metric signal.
## No Alerting
- Health monitoring is **display-only** for now — no automated notifications or alerts are triggered by health status changes.
@@ -113,6 +234,7 @@ These tiles are **point-in-time** like the Notification Outbox and Site Call Aud
- **Site Runtime (site)**: Provides script error rate and alarm evaluation error rate metrics.
- **Store-and-Forward Engine (site)**: Provides buffer depth metrics, including the notification backlog awaiting forward to central.
- **Cluster Infrastructure (site)**: Provides node role status.
- **Host (site)**: Supplies the two boot-time facts Health Monitoring must not derive for itself — the LocalDb replication-configured gate behind `LocalDbReplicationConnected`/`LocalDbOplogBacklog` (from `SiteLocalDbSetup.ReplicationIsConfigured`), and `StartupReconcilerActive`, latched where the Host decides whether to spawn `SiteReconciliationActor`. Both follow the `SiteEventLogReplicationCheck` precedent: the Host owns the configuration reading, the component owns the reporting.
- **Notification Outbox (central)**: Provides central-computed outbox KPIs — queue depth, stuck count, parked count — for the headline dashboard tiles.
- **Site Call Audit (central)**: Provides central-computed cached-call KPIs — buffered count, parked count, failed/delivered (last interval), oldest pending age, stuck count — for the headline dashboard tiles.
- **Site Event Logging (site)**: Provides `SiteEventLogWriteFailures`. Its daily retention purge and storage-cap trim are also the expected cause of transient `LocalDbOplogBacklog` spikes on a replicated node — see [Component-SiteEventLogging.md](Component-SiteEventLogging.md) → Storage.
@@ -122,4 +244,4 @@ These tiles are **point-in-time** like the Notification Outbox and Site Call Aud
- **Central UI**: Health Monitoring Dashboard displays aggregated metrics.
- **Communication Layer**: Health reports flow as periodic messages.
- **KPI History (#26)**: emits `IKpiSampleSource` (`SiteHealthKpiSampleSource`, per-Site) consumed by the KpiHistory recorder (#26). It reads the in-memory `ICentralHealthAggregator.GetAllSiteStates()` (no DB), turning the per-site snapshot — previously sequence-numbered every 30s but discarded — into 12 sampled `KpiSample` metrics per site, persisted every minute: `connectionsUp`/`connectionsDown`, `scriptErrors`, `alarmEvalErrors`, `sfBufferDepth`, `deadLetters`, `parkedMessages`, `deployedInstances`/`enabledInstances`/`disabledInstances`, `auditBacklogPending`, `eventLogWriteFailures`. Of these, **only four are charted** in the dashboard's per-site `KpiTrendChart` panel — `connectionsDown`, `scriptErrors`, `sfBufferDepth`, and `deadLetters` (the metrics carried in the shared `KpiMetrics.SiteHealth` Commons catalog and fetched by `Health.razor`'s trend selector). The remaining eight are **sampled and persisted but not (yet) charted** — they are retained in the `KpiSample` store and available for future trend surfaces or ad-hoc query. The synthetic central self-report (`CentralHealthReportLoop.CentralSiteId = "$central"`, fed by the leader-only report loop via `CollectReport($central)`) appears in `GetAllSiteStates()` and is **intentionally sampled into the per-Site KPI store as a real `KpiScopes.Site` series** keyed `$central`; the Central UI deliberately **pins `$central` first** in the per-site trend selector (rendered as "Central Cluster"), so the central cluster's own trends sit alongside the real sites'. (`$central`'s zero-valued connection/instance/S&F metrics reflect the central self-report carrying no site-runtime data, not a defect.) See [Component-KpiHistory.md](Component-KpiHistory.md).
- **KPI History (#26)**: emits `IKpiSampleSource` (`SiteHealthKpiSampleSource`, per-Site) consumed by the KpiHistory recorder (#26). It reads the in-memory `ICentralHealthAggregator.GetAllSiteStates()` (no DB), turning the per-site snapshot — previously sequence-numbered every 30s but discarded — into 12–14 sampled `KpiSample` metrics per site, persisted every minute: `connectionsUp`/`connectionsDown`, `scriptErrors`, `alarmEvalErrors`, `sfBufferDepth`, `deadLetters`, `parkedMessages`, `deployedInstances`/`enabledInstances`/`disabledInstances`, `auditBacklogPending`, `eventLogWriteFailures`, plus the two conditionally-emitted LocalDb replication metrics `localDbOplogBacklog` and `localDbReplicationConnected` (register row 33 — emitted only when the site actually reports a reading, hence 12 for an unreplicated site and 14 for a replicated one; see *KPI History Sampling of Replication State* above). Of these, **only five are charted** in the dashboard's per-site `KpiTrendChart` panel — `connectionsDown`, `scriptErrors`, `sfBufferDepth`, `deadLetters`, and `localDbOplogBacklog` (the metrics carried in the shared `KpiMetrics.SiteHealth` Commons catalog and fetched by `Health.razor`'s trend selector). The rest are **sampled and persisted but not (yet) charted** — they are retained in the `KpiSample` store and available for future trend surfaces or ad-hoc query. The synthetic central self-report (`CentralHealthReportLoop.CentralSiteId = "$central"`, fed by the leader-only report loop via `CollectReport($central)`) appears in `GetAllSiteStates()` and is **intentionally sampled into the per-Site KPI store as a real `KpiScopes.Site` series** keyed `$central`; the Central UI deliberately **pins `$central` first** in the per-site trend selector (rendered as "Central Cluster"), so the central cluster's own trends sit alongside the real sites'. (`$central`'s zero-valued connection/instance/S&F metrics reflect the central self-report carrying no site-runtime data, not a defect.) See [Component-KpiHistory.md](Component-KpiHistory.md).
docs/requirements/Component-Host.md
+66
View File
@@ -181,6 +181,72 @@ Each component library must expose its services to the Host via a consistent set
The Host's `Program.cs` calls these extension methods; the component libraries own the registration logic. This keeps the Host thin and each component self-contained. (The ManagementService component no longer registers the ManagementActor with ClusterClientReceptionist — that registration was removed 2026-07-22, and the whole receptionist surface went in Phase 4.)
### REQ-HOST-11: Dependency-Injection Lifetime Discipline
Akka actors are constructed by the actor system, not by the DI container, so the container cannot enforce lifetimes on them. Three rules therefore apply, and each has a mechanical guard.
**1. An actor may hold the root `IServiceProvider` (or an `IServiceScopeFactory`) for its lifetime — never a scope or anything resolved from one.** Every operation that needs a scoped service opens its own scope (`CreateScope` / `CreateAsyncScope`, disposed in the same handler) and resolves inside it. Caching the resolved instance in a field is a **captive dependency**: the actor keeps one EF `DbContext` alive for its whole lifetime and any concurrency the actor allows then collides on it. This is not hypothetical — it shipped as register row 45, where `NotificationOutboxActor` cached delivery adapters from an actor-lifetime scope and `MaxParallelDeliveries` concurrent deliveries drove EF's "a second operation was started on this context instance" continuously on the rig.
*Guard:* `CaptiveDependencyGuard` (Host test suite) reflects over every actor class in the shipped assemblies and fails on any instance field — auto-property backing fields included — typed as `IServiceScope`, as a `DbContext`, or as a service the real composition roots register `Scoped`. The scoped set is derived from the actual registrations of the built Central and Site containers rather than hand-listed, so changing a registration's lifetime automatically re-scopes the check. `IServiceProvider` / `IServiceScopeFactory` fields are explicitly permitted. Exemptions live in a small allowlist inside the guard, each carrying its safety argument, and a self-test fails any allowlist entry that has stopped matching a real field.
**2. Scope validation is on in Development, for both roles.** `Program.cs` applies `ValidateScopes` + `ValidateOnBuild` via `ServiceProviderValidation` in both the Central and Site branches. This is the container-visible half of the same concern — it catches a singleton whose *constructor* takes a scoped service, which is the ordinary form of the bug in non-actor code. It cannot see rule 1's hand-rolled `GetRequiredService` calls, and rule 1's guard cannot see constructor injection; neither subsumes the other. Production runs unvalidated deliberately: a validation failure is fail-closed, and a defect must not take a production node down at boot when Development and CI give the same signal.
**3. A registration must be constructible on every role that registers it.** A service whose dependencies exist only on the other role belongs in a role-scoped extension method (`AddCentralCommunication`, `AddAuditLogCentralKpi`), not in a shared `AddXxx()` on the argument that "only the central actor ever resolves it". An unconstructible descriptor is not inert: it fails `ValidateOnBuild` for the *entire* graph, which is what previously kept rule 2 switched off on the Site branch and would have masked any genuine defect the validator found there.
---
### Boot-time facts the Host supplies to Health Monitoring
Health Monitoring reports two things the Host alone knows, and in both cases the Host reads the
configuration and hands over a captured value rather than teaching the component to read it. This
is the `SiteEventLogReplicationCheck` pattern generalised: configuration-reading rules live in one
place, and a component that merely *reports* a fact never grows a dependency on where it comes
from.
**LocalDb replication configured-ness** (`SiteServiceRegistration`, register row 80). The
replication engine is registered unconditionally and idles without a peer, so `ISyncStatus` answers
`Connected = false` with a real `0` backlog on a node that was never meant to replicate. Passing
that through made every deliberately-unreplicated site publish "disconnected, empty backlog". The
Host now evaluates `SiteLocalDbSetup.ReplicationIsConfigured(config)` **once** and passes it into
`AddLocalDbReplicationHealthBridge` as a third provider; an unconfigured node publishes `null` for
both LocalDb health fields. The same captured value feeds `SiteEventLogReplicationCheck`, so the
two surfaces can never disagree about whether this node replicates. Note the rule is `PeerAddress`
**OR** `ApiKey` — only the dialling half sets a peer address.
**Startup reconciler presence** (`AkkaHostedService`, register row 73). The Host decides whether to
spawn `SiteReconciliationActor` — it needs an `IDeploymentConfigFetcher`, and skips the actor when
none is resolvable. That skip was previously a lone boot Warning, so a node with no self-heal path
at all reported itself perfectly healthy. Both branches now call
`ISiteHealthCollector.SetStartupReconcilerActive(...)`, latching the fact onto every subsequent
health report as tri-state `StartupReconcilerActive`; the skip branch additionally logs at **Error**
(raised from Warning — no configuration makes the fetcher legitimately absent on a site node) and
writes a best-effort `deployment`/`Error` site event so the evidence outlives the boot log. Central
renders it as a *No self-heal* warning badge. See
[Component-HealthMonitoring.md](Component-HealthMonitoring.md) → *Startup Reconciler Presence*.
### Standby health-check log noise
`LoggerConfigurationFactory` installs one Serilog exclusion filter, for the by-design standby
verdict of the `active-node` probe (register row 71).
`DefaultHealthCheckService` logs one line per check per probe and picks the level from the verdict —
Debug / Warning / **Error** for Healthy / Degraded / Unhealthy. Both active-node checks
(`SitePairActiveNodeHealthCheck` on a site, the shared `ActiveNodeHealthCheck` on central) answer
Unhealthy on the standby, because 503 is exactly the contract `/health/active` publishes to Traefik
and the family dashboard. A perfectly healthy standby therefore logged ERR on every probe — every
10 s on a site node, every 5 s on a central node behind Traefik's active-node probe — which floods
the log and trains operators to ignore ERR.
Serilog's `MinimumLevel.Override` can only *raise* a per-source floor and a `LogEvent`'s level is
immutable, so neither an override nor an enricher can rewrite the event; excluding it is the only
surgical option. The predicate matches on `Level == Error` **and** the health-check logger category
**and** `HealthCheckName == "active-node"`. Keying on the check name rather than the category is
what preserves the constraint: a failing `localdb`, `akka-cluster`, `database` or
`required-singletons` check still logs at Error, and the active-node check has no error path of its
own (the active node always answers Healthy), so the only event this can ever suppress is a standby
reporting that it is a standby. The verdict remains fully observable through the HTTP status of
`/health/active` itself, the readiness probe's cluster view, and the health dashboard.
---
## Component Registration Matrix
docs/requirements/Component-KpiHistory.md
+11
View File
@@ -143,6 +143,17 @@ Reads `ICentralHealthAggregator.GetAllSiteStates()` (in-memory, no DB). Scope: *
- `disabledInstances`
- `auditBacklogPending`
- `eventLogWriteFailures`
- `localDbOplogBacklog` — **conditionally emitted** (register row 33)
- `localDbReplicationConnected` — **conditionally emitted** (register row 33), 0/1
The last two are written **only when the site actually reports a reading**. Both underlying health-report
fields are nullable and null means "no reading" (replication not configured, or a failed poll), never
zero — so unlike `auditBacklogPending` above, they are not coerced with `?? 0`. A site therefore emits
**12 metrics when unreplicated and 14 when replicated**, and a gap in these two series truthfully means
"not replicated / not reporting" rather than a flat zero line that would render an unreplicated pair, or
one that cannot read its own oplog, as perfectly healthy. Both fold as **Gauge** in
`KpiMetricAggregationCatalog` (they are instantaneous readings; summing an oplog depth or a 0/1 flag per
hour would be meaningless). `localDbOplogBacklog` is charted on the Health dashboard's per-site trend panel.
## Query + UI
docs/requirements/Component-SiteEventLogging.md
+72 -3
View File
@@ -93,19 +93,88 @@ Each event entry contains:
rig's site-b/site-c), the log stays node-local and a failover does start it fresh — the
documented trade of not configuring a peer, unchanged by this policy.
- **Retention**: 30 days. A **daily background job** runs on the active node and deletes all events older than 30 days. Hard delete — no archival. Deletes are sliced into bounded 1000-row batches per DELETE statement rather than one unbounded statement, so a large expired backlog does not hold the write lock (and every concurrent recorder flush) for the duration of the purge.
- **Storage cap**: A configurable maximum database size (default: 1 GB) is enforced. If the storage cap is reached before the 30-day retention window, the oldest events are purged first. This prevents disk exhaustion from alarm storms, script failure loops, or connection flapping.
- **Storage cap**: A configurable maximum size (default: 1 GB) is enforced. If the cap is reached
before the 30-day retention window, the oldest events are purged first. This prevents disk
exhaustion from alarm storms, script failure loops, or connection flapping.
- **The cap measures `site_events` and its four indexes — NOT the database file** (register row 81;
decision record `docs/plans/2026-08-17-site-event-log-cap-semantics-decision.md`). This matters
because the event log shares one SQLite file with nine other tables plus LocalDb's
`__localdb_oplog` / `__localdb_row_version` / `__localdb_snapshot_state` bookkeeping. The cap used
to compare `MaxStorageMb` against `page_count − freelist_count` over that whole file, which on a
replication-enabled node made the trim **unwinnable**: each 1000-row delete freed ~390 KB of event
pages while adding ~1000 oplog rows and ~1000 tombstones to the same file, the measured size did
not fall, and the progress guard stopped the trim — measured 2.5x over a 64 MB cap with 226 k rows
still present. A node whose peer was down could not enforce its cap at all. Two consequences of
the new scoping are worth internalising:
- **The cap is now independent of replication health.** It was previously a function of the
*peer's* liveness, and in the wrong direction — an inflated oplog is a replication fault, and
letting it drive event deletion would destroy operational history in response to one.
- **`MaxStorageMb` no longer bounds the data volume on its own.** LocalDb bookkeeping is separate
overhead on the same disk, bounded separately by `LocalDb:Replication:MaxOplogRows` and
`TombstoneRetention`; sizing a site's volume means adding the two. Managing that bookkeeping is
deliberately **not** this component's job — see the decision record §6.
- **The measurement is modelled, not exact.** SQLite can report per-table page usage precisely via
the `dbstat` virtual table, and that was the intended fix — but `dbstat` is a compile-time option
and the bundled `SQLitePCLRaw.lib.e_sqlite3` native library (SQLite 3.53.3) does **not** enable it;
neither `dbstat` nor `sqlite_dbpage` exists, verified by probe and pinned by
`SqliteCapabilityTests`. The table's footprint is therefore estimated from sampled row width
(500 rows from each end of the table, measured with `octet_length` so multi-byte UTF-8 is not
under-counted) fed into a page-layout model; index pages count against the cap, deliberately,
because they are ~34 % of the table's real footprint and the 1 GB default was sized against a
bytes/row figure that included them. Measured error is **−0.8 % to +12.8 %** across representative
workloads, biased high on purpose — a cap that fires slightly early forfeits budget, whereas one
that reads low fails at the only job it has. The estimate is additionally clamped by the
whole-file logical size, which is a true upper bound on any one table.
- **Two wipe-prevention backstops** guard the cap trim, replacing the single "on-disk size did not
decrease" check. (1) If row width cannot be sampled, the trim **refuses to run** and warns —
checked before the cap comparison, because a zero row width makes "are we over cap" unanswerable
rather than false, and the old loop's answer to an unanswerable cap was to keep deleting. (2) If
the *estimated table size* fails to fall across a batch, the trim stops and warns. Guard (2) is the
old check re-based onto the table-scoped metric, and its meaning is now unambiguous: replication
bookkeeping can no longer trip it, so it fires only when new events are arriving at least as fast
as the trim removes them. For that to be observable, the loop re-reads the row count from the
database after every batch, while bytes-per-row is sampled **once per tick and held fixed** —
re-sampling per batch would make the estimate jitter as the trim eats the head of the table and
could stall the loop through guard (2), which is the same self-defeating-measurement failure as
row 81 itself.
- **Per-tick delete ceiling** (`MaxPurgedRowsPerTick`, default **250 000**; `0` = unlimited): one
purge tick deletes at most this many rows, **shared** by the retention purge and the storage-cap
trim — two independently-capped phases would let a tick emit twice the intended burst. A tick that
stops at the ceiling schedules the next tick after `PurgeCatchUpInterval` (default **5 minutes**)
instead of `PurgeInterval`, so retention still converges: a worst-case backlog drains over
~11 accelerated ticks (≈55 minutes) rather than 11 days at the daily cadence, with every
individual burst bounded. The ceiling exists because the purge is CDC-captured (next bullet) and
an unbounded tick could exceed LocalDb's oplog budget; measured evidence and the chosen defaults
are in `docs/plans/2026-08-15-site-events-purge-burst-measurement.md`. A ceiling-limited tick logs
a Warning (rate-limited to one line per 15 minutes) and increments
`scadabridge.site.event_log.purge_ceiling_hits` — the counter is the unthrottled series that
distinguishes a backlog converging from a site whose event rate outruns one paced tick.
Consequence of the accelerated retry: the purge loop is a delay loop, not a `PeriodicTimer`, so
the cadence is anchored to the **end** of the previous purge rather than to process start.
- **Purge deletes are CDC-captured on a replicated site — the resulting backlog spike is expected.**
Retention and cap deletes are ordinary row changes on a replicated table; there is deliberately no
purge-exemption path (LocalDb Phase 2 — CDC does all three jobs, and a table-wide exemption is the
same mechanism that made `ReplaceAllAsync` unsafe). One oplog row is queued per deleted event, so
the site health report's `LocalDbOplogBacklog` (Prometheus `localdb_oplog_depth`) jumps by the size
of the batch at purge time and drains as the peer acks it. **Healthy signature:**
of the batch at purge time and drains as the peer acks it. **The height of that jump is bounded by
`MaxPurgedRowsPerTick`, not by how much history expired** — measured (register row 34): a full 1 GB
event log holds ~2.73 million rows at 393 bytes/row, and purging it unbounded queued 2 731 947
oplog rows, **2.7x LocalDb's `MaxOplogRows` default of 1 000 000**, which would make the
maintenance tick prune the oplog and flag a full ten-table snapshot resync. How much of that is
*resident* depends on the peer — the oplog drains concurrently, and a loopback peer held the same
burst to a 391 000 peak — so the overrun is conditional on peer throughput and unconditional when
the peer is down. Bounding the tick is what makes the outcome independent of the peer. Pacing the
tick, not exempting the purge from capture, is the fix — the exemption remains ruled out.
**Healthy signature:**
`LocalDbReplicationConnected` stays true across the spike and the backlog returns to ~0; a backlog
that keeps climbing, or climbs while `LocalDbReplicationConnected` is false, is a genuine
replication fault and not the purge. The purge logs an Information line naming the row count and
the expected transient backlog whenever it deletes rows on a replication-enabled node, so an
operator can correlate a spike with the purge that caused it — a spike with no such line nearby is
not the purge. Node-local wiring: the Host supplies the replication predicate
not the purge. That line is emitted **once per tick**, carrying the retention purge's and the cap
trim's deletes combined (register row 84 — it used to fire once per phase, so a tick in which both
deleted produced two identical notes; the tick total is also the more useful number to put beside
an oplog spike). Node-local wiring: the Host supplies the replication predicate
(`SiteLocalDbSetup.ReplicationIsConfigured` — the `PeerAddress`-OR-`ApiKey` rule) as
`SiteEventLogReplicationCheck`; this component never reads LocalDb configuration itself, and with
no predicate registered the note is suppressed. Operator detail —
docs/requirements/Component-SiteRuntime.md
+16
View File
@@ -96,6 +96,20 @@ flowchart TD
> **Startup config load is retried (S5)**: step 1's deployed-config read is best-effort against local SQLite, which can transiently fail (locked/busy). A failed load must **not** leave the site as a silent zero-instance node — it is re-attempted on a fixed interval until it succeeds, then the retry self-cancels.
> **Startup reconcile heals the RUNTIME, not just the config file (register row 67)**: `SiteReconciliationActor` runs on **every** site node (not a singleton) and performs one best-effort pass shortly after boot — it reports the node's local deployed inventory to central and fetches + stores any config central reports as a gap. That pass fires **after** step 1's deployed-config read has already run, so storing a config started nothing: Instance Actors are created only by the boot-time load above and by an explicit deploy/enable. An instance whose config was missing at boot (lost volume, first boot after a restore, rehomed data dir) therefore stayed **dead until a second restart**, whose load finally picked it up — silently, with nothing surfacing "config stored, actor not running". The pass now raises a fire-and-forget `EnsureInstanceStartedCommand` for every **enabled** instance it healed, routed through the **deployment-manager singleton proxy** so Instance Actors are only ever created under the singleton regardless of which node's pass found the gap. The handler is deliberately advisory: it **no-ops when the actor is already running** (both nodes legitimately raise it for the same instance), **re-reads the durable row** — which is the authority on both the config body and enabled/disabled state, so a disable landing after the fetch still wins and the instance stays stopped — **defers to any in-flight deploy/redeploy** for that instance, preserving the deploy gate's ordering guarantees and the all-or-nothing-per-instance rule (it never applies a config, only starts one already fully persisted), and treats a missing row as a quiet no-op (the normal case on an **unreplicated** site whose peer node healed its own file). A successful reconcile start is **surfaced**, not silent: a Warning log line plus an operational `deployment` site event, because reaching it means the node booted without a config for an instance that should have been running.
> **Instance reconcile sweep — the store and the runtime are kept in agreement, and disagreement is surfaced (register rows 78 + 72)**: the reconcile self-heal above covers configs the reconcile pass *itself* fetched. Two sibling divergences were not covered, and neither surfaced anything.
> - **Row 78 — a config that arrives by replication starts nothing.** `deployed_configurations` is one of LocalDb's ten replicated tables, so on a replicated pair a config can appear in a node's file with **no local actor involved at all** — the CDC sync applier writes the row directly. Proven live 2026-08-15: a fresh site-a-a was seeded from its peer by snapshot replication within ~5 s of boot, i.e. **after** its boot-time load had read an empty table and **invisibly** to its own reconcile (central rightly reported no gap, because central's view of the site was correct). Enabled instances sat dead with their configs present until another restart.
> - **Row 72 — a refreshed config does not reach the running actor.** For a *stale* `ReconcileGapItem` the pass writes the newer config via `StoreDeployedConfigIfNewerAsync`, a pure file write; the running Instance Actor keeps executing the previously compiled config until the next restart or redeploy.
>
> Both are closed by one **periodic sweep owned by the Deployment Manager singleton** (`ScadaBridge:SiteRuntime:InstanceSweepIntervalSeconds`, default 60, validated `>= 0`; `0` disables it). Each tick re-reads the durable rows off-thread and, on the actor thread, for every **enabled** stored config: starts the Instance Actor if none is running (row 78), or — if one is running at a revision the store has since replaced — **re-applies the stored config through the exact per-instance apply/restart path a central redeploy uses** (row 72). A sweep is chosen over a LocalDb change hook deliberately: a hook would cover only the replication arrival path, only on replicated sites, and `ILocalDb` exposes no change-notification API at all, whereas the sweep is arrival-path-agnostic (replication seeding, post-boot snapshot resync, a hand-restored file, any future path). See `docs/plans/2026-08-15-instance-sweep-self-heal-design.md`.
>
> The **row-72 decision is to converge, not merely warn**: the stored row *is* what central successfully deployed, so an actor serving the config central believes was replaced is the actual fault. Reusing `HandleDeploy` means the compile gate, warm-then-gate ordering, redeploy-supersede, the static-override / native-alarm reset and all-or-nothing-per-instance are the production ones — and a stop still does **not** cancel an in-flight script run, exactly as for a central redeploy.
>
> Guards, all shared with the row-67 handler: a **disabled** instance is never started; the sweep **defers to any deploy pipeline that owns the instance** (compile warm in flight, predecessor still terminating, deployment awaiting its persistence/init join) so it can never interleave with a real redeploy; it is **inert until boot-time Instance Actor creation has finished**, so a tick between staggered batches does not "heal" instances the boot load is about to create; it makes **one repair attempt per (instance, revision) pair**, which is what stops a periodic self-heal becoming a restart loop for an actor that dies during init or a stored config the compile gate rejects (a rejection leaves the running instance untouched — the gate returns before anything is stopped); and reads never overlap. A sweep that finds nothing is **completely silent**. A repair emits the same Warning `deployment` site-event pattern as row 67, naming the trigger (`deployment-manager sweep self-heal`) so it is distinguishable from a startup-reconcile heal, and a stale-revision restart names **both** revisions; a rejected restart adds an Error event. The synthetic redeploy replies to the singleton itself rather than to `NoSender`, because dead letters are a monitored site health metric.
> **Starting an Instance Actor also creates its DCL connections — the ensure lives in the creation funnel, not at call sites (register row 85)**: a DCL connection actor is **never** created from the `data_connection_definitions` table (nothing in production reads it). It exists only as a side effect of starting an instance, built from the `Connections` map inside that instance's **own flattened config** via `EnsureDclConnections` — plus the separate system-wide artifact-deploy path. That ensure used to be each *caller's* responsibility, and only two of the four callers made it (the boot-time staggered batch and `ApplyDeployment`). The two that did not were **`StartStoredInstance`** — the path shared by the row-67 ensure-started handler and the row-78 sweep — and **`HandleEnableResult`**. Both therefore produced a running instance whose every tag subscribe was answered `Unknown connection: <name>` by the DCL manager, indefinitely, because nothing else on the node would ever create that connection actor. Proven live 2026-08-16: a replication-seeded node's instances were correctly started by the sweep and then failed every subscribe for 6+ minutes until a second pair restart let the boot load create the connections. **It was never replication-specific** — the boot-time load reads only *enabled* configs, so a node that boots with an instance disabled never creates its connections either, and enabling it afterwards hit the identical dead end on an ordinary unreplicated site. `EnsureDclConnections` now sits at the top of **`CreateInstanceActor`**, already the single Instance Actor creation funnel (it is where the running revision is stamped for the same reason), so all four paths get it and a fifth cannot forget. It is placed *before* the already-exists guard so the two paths that already called it keep their exact previous semantics, and it stays silent in steady state because `EnsureDclConnection` is hash-guarded on the last config pushed per connection name. Deliberately **not** done: a sweep pass over `data_connection_definitions` — it would not have fixed row 85 (that table is empty on a site that has only ever received instance deploys), and re-hashing from it would fight the deploy path over the same per-name hash slot, recreating connection actors and dropping live subscriptions on a loop. See `docs/plans/2026-08-15-instance-sweep-self-heal-design.md` §7.
> **Per-instance compilation during staggered startup — CLOSED (WP3.1)**: this was a deferred optimization (P6) affecting failover time-to-recover; each Instance Actor Roslyn-compiled its own scripts inside its own start, serialising a site's whole recovery behind compilation. Each staggered startup batch now runs a **pre-warm step**: before the batch's Instance Actors are created, the Deployment Manager compiles that batch's distinct script bodies and trigger expressions on a background task and pipes `BatchCompileWarmed` back, so every `PreStart` compile is a cache hit. Instance Actor start still calls the compilation service — it remains the correctness backstop — but in every warmed path those calls are memoised lookups. Cost: one extra message per batch.
> **The Roslyn `ScriptOptions` are process-static, and must stay that way**: the reference set backing every site-side compile is built once per process, never per compile. `ScriptOptions.WithReferences(Assembly[])` resolves each assembly through `MetadataReference.CreateFromFile`, which does **not** cache — each call mints a fresh `MetadataReference` owning an `AssemblyMetadata` → `PEReader` → `NativeHeapMemoryBlock`, an unmanaged copy of the assembly metadata that nothing disposes. Building the options per compile leaks native memory permanently: no GC reclaims it, and it is invisible to gcdump and to the managed allocation counters, so the node's working set grows without the GC heap growing. This was a live defect (a Site node at 2,885 MB working set with 150 MB of live GC heap, ~2,469 MB of it on the default process heap across ~6,700 undisposed `AssemblyMetadata` instances). Note this is orthogonal to the compile cache above — the cache dedupes *identical* script bodies, so it bounds nothing when the bodies differ. (It no longer clears wholesale on overflow: WP3.1 replaced that latency cliff with approximate-LRU batch eviction, see **Compile cache eviction** below.) Pinned by a reference-equality regression test rather than by watching memory.
@@ -124,6 +138,8 @@ flowchart TD
- **Disable**: Stops the Instance Actor and its children. Retains the deployed configuration in SQLite so the instance can be re-enabled without redeployment.
- **Enable**: Creates a new Instance Actor from the stored configuration (same as startup).
- **Delete**: Stops the Instance Actor and its children, removes the deployed configuration from local SQLite. Does **not** clear store-and-forward messages.
- **Ensure Started** (`EnsureInstanceStartedCommand`, site-internal — never sent by central): advisory, idempotent request to make sure an already-persisted, enabled instance is actually running. Raised by the startup reconcile self-heal; see the note under **Startup Behavior**. Unlike the three commands above it has **no reply** and applies no configuration — it only starts an Instance Actor for a config that is already fully stored, and no-ops when the actor is running, the stored row says disabled, the row is absent, or a deploy for that instance is in flight.
- **Instance reconcile sweep** (internal timer, not a command): the periodic singleton-owned check that the running Instance Actors still agree with the durable config rows — starting an enabled instance that has none, and re-applying a stored config a running instance has fallen behind. See the note under **Startup Behavior**; the start half shares its implementation (and therefore its bookkeeping and site events) with **Ensure Started**, and the restart half goes through the ordinary redeploy path.
### Debug Snapshot Routing
- Receives `DebugSnapshotRequest` from the Communication Layer and forwards to the Instance Actor by unique name (same lookup as `SubscribeDebugViewRequest`).
docs/requirements/Component-StoreAndForward.md
+13
View File
@@ -63,6 +63,19 @@ For the external-system-call and cached-database-write categories, retry setting
- **External systems**: Each external system definition includes max retry count and time between retries.
- **Cached database writes**: Each database connection definition includes max retry count and time between retries.
Both are carried to the site in the deployment artifact and read back out of the site's own
tables by the gateway, which passes them to `EnqueueAsync`. An **unset** source-entity value
(`MaxRetries` `0` / `RetryDelay` `TimeSpan.Zero`) is passed as `null`, so the engine's
`DefaultMaxRetries` (50) and `DefaultRetryInterval` (30 s) apply — roughly 25 minutes to park.
The translation is deliberate and must not be removed: the engine reads a stored `MaxRetries`
of `0` as **no limit**, so forwarding an unconfigured entity's `0` verbatim would turn every
cached call into an unbounded retry loop.
> Until deferred-work row 27 was fixed, this was true of database connections only.
> `ExternalSystemArtifact` carried no retry fields and the site `external_systems` table had no
> columns for them, so an external system's centrally-configured policy never reached the site
> and every external-system cached call silently used the 50 x 30 s default.
The **notification** category retries differently: it has no source-entity setting. The site→central forward uses a single fixed retry interval configured in the host `appsettings.json`. This interval is infrastructure config for reaching the central cluster, not a per-notification-list setting. It applies uniformly to every buffered notification regardless of its target list. A buffered notification is retried at that interval until central acks it; the engine's `DefaultMaxRetries` cap still applies (matching the cached-call categories) and a notification whose retries are exhausted under a sustained central outage parks like any other buffered message. The cap is sized so the normal central-recovery window stays well inside it — central, once reachable, owns delivery, retry, and parking from the ack point on.
The retry interval is **fixed** (not exponential backoff). Fixed interval is sufficient for the expected use cases.
docs/requirements/Component-Transport.md
+4 -2
View File
@@ -134,7 +134,7 @@ flowchart TD
class T muted
```
The component is central-hosted. It is registered in `ZB.MOM.WW.ScadaBridge.Host` for central roles only, never for site roles — even when a bundle carries site-scoped config (T18), Transport only writes the central configuration database, never the site clusters. All persistence flows through existing audited repository interfaces in `ZB.MOM.WW.ScadaBridge.ConfigurationDatabase` — the component does not call `DbContext.SaveChangesAsync` directly. `BundleSessionStore` is in-process on the active central node (matching Blazor Server circuit affinity): 30-minute TTL, eviction on expiry, 3-strike passphrase lockout per session, and a `MaxConcurrentImportSessions` cap (default 8; arch-review 05 Task 24 — startup-validated as `> 0` by `TransportOptionsValidator`, since a zero/negative cap would reject every import session forever) bounding the decrypted-content footprint since each open session pins a fully-decrypted bundle in memory.
The component is central-hosted. It is registered in `ZB.MOM.WW.ScadaBridge.Host` for central roles only, never for site roles — even when a bundle carries site-scoped config (T18), Transport only writes the central configuration database, never the site clusters. All persistence flows through existing audited repository interfaces in `ZB.MOM.WW.ScadaBridge.ConfigurationDatabase` — the component does not call `DbContext.SaveChangesAsync` directly. `BundleSessionStore` is in-process on the active central node (matching Blazor Server circuit affinity): 30-minute TTL, eviction on expiry, 3-strike passphrase lockout per session, and a `MaxConcurrentImportSessions` cap (default 8; arch-review 05 Task 24 — startup-validated as `> 0` by `TransportOptionsValidator`, since a zero/negative cap would reject every import session forever) bounding the decrypted-content footprint since each open session pins a fully-decrypted bundle in memory. **The cap is observable (register row 53):** open (unexpired) sessions are published as the `scadabridge.central.transport.import_sessions.open` gauge on the `ZB.MOM.WW.ScadaBridge` meter — already in the `ObservedMeters` allowlist, so no telemetry config change is needed — and a rejection at the cap logs at Warning with the current count, the cap and the TTL. Reclaiming a store full of *expired* sessions is not a cap hit and logs nothing. **A log line is not alertable, so register row 66 adds the counter `scadabridge.central.transport.import_sessions.rejected`** on the same meter, incremented at the same single genuine-rejection point (the post-eviction recheck — never on the reclaim-then-succeed path, and never when re-opening an already-open session id). Alert on the counter; interpret it against the gauge — rejections rising while the gauge stays below the cap means genuine concurrency (raise the cap), whereas rejections rising while the gauge is pinned at the cap means a leaked session (raising the cap only defers it).
**Session release is explicit, not TTL-only (register row 43).** `ApplyAsync` releases the session it applied (zero-then-remove, T-007), and `IBundleImporter.Abandon(sessionId)` does the same for a session that will never be applied. Every abandonment path calls it: the Central UI wizard on new upload / Back-to-Upload / `ResetSessionState` / lockout reset / `Dispose` (navigation away or circuit teardown), the CLI `transport preview` handler (a preview never applies), and the CLI `transport import` handler for the aborts that precede apply (blocker rows, unresolved mappings). `Abandon` is idempotent — double release, an unknown id, and an already-expired id are all silent no-ops — and since session ids are random GUIDs handed only to their opener, abandoning the id you hold can never evict another operator's in-progress import. Relying on the TTL alone was an availability *and* secret-lifetime defect: eight abandoned wizards exhausted `MaxConcurrentImportSessions` and locked out every import on the active node for 30 minutes, each still holding its decrypted bundle content.
@@ -336,7 +336,9 @@ Imports are all-or-nothing per bundle. A bundle either applies fully or not at a
- **AES-256-GCM** for content encryption with **PBKDF2-SHA256 / 600 000 iterations** (OWASP 2023+ guidance), per-bundle random salt, random IV per encryption. The GCM auth tag is verified before decryption — a wrong passphrase fails cleanly.
- **Passphrase never persisted.** It lives only inside the export/import service call path and is discarded after use. No environment variable, no log line.
- **Failed-unlock rate limit:** per-bundle 3-strike lockout (keyed by `ContentHash`, server-side); per-IP-per-hour cap (`MaxUnlockAttemptsPerIpPerHour`, default 10, configurable) to deter brute force against a stolen bundle. Each failed attempt produces a `BundleImportUnlockFailed` audit row. The two are evaluated in order — bundle lockout, then per-IP throttle, then decrypt — so hammering an already-locked bundle spends no hourly budget, and a throttle denial (`BundleUnlockThrottledException`, fail-closed with a curated message on both the UI and CLI surfaces) burns **no** per-bundle strike, since it never reaches the decryptor. Successful unlocks count against the hourly budget too: it caps attempts, not failures. The client key is the caller's remote IP — from `CircuitClientAddressProvider` (snapshotted at Blazor circuit establishment, since `HttpContext` is unavailable mid-circuit) for the Central UI, and from `ManagementEnvelope.ClientIp` for the CLI; a caller with no attributable address shares a single stricter bucket rather than bypassing the throttle. Per-client isolation is only real behind a ForwardedHeaders-trusted proxy, the same caveat the login throttle carries in `Component-Security.md`. (Register row 44: the limiter existed, was configured and unit-tested, but was never invoked until this was wired.)
- **Failed-unlock rate limit:** per-bundle 3-strike lockout (keyed by `ContentHash`, server-side); per-IP-per-hour cap (`MaxUnlockAttemptsPerIpPerHour`, default 10, configurable) to deter brute force against a stolen bundle. Each failed attempt produces a `BundleImportUnlockFailed` audit row. The two are evaluated in order — bundle lockout, then per-IP throttle, then decrypt — so hammering an already-locked bundle spends no hourly budget, and a throttle denial (`BundleUnlockThrottledException`, fail-closed with a curated message on both the UI and CLI surfaces) burns **no** per-bundle strike, since it never reaches the decryptor. Both typed exceptions live in `Commons/Types/Transport/` — `ManagementService` references Commons but not Transport, so a type declared beside the importer cannot be curated by `MapFault` and reaches the CLI as a generic internal-error message (register rows 44 and 51). Successful unlocks count against the hourly budget too: it caps attempts, not failures. The client key is the caller's remote IP — from `CircuitClientAddressProvider` (snapshotted at Blazor circuit establishment, since `HttpContext` is unavailable mid-circuit) for the Central UI, and from `ManagementEnvelope.ClientIp` for the CLI; a caller with no attributable address shares a single stricter bucket rather than bypassing the throttle. Per-client isolation is only real behind a ForwardedHeaders-trusted proxy, the same caveat the login throttle carries in `Component-Security.md`. (Register row 44: the limiter existed, was configured and unit-tested, but was never invoked until this was wired.)
**The per-IP window is fleet-wide and restart-durable (register row 52).** It is persisted in the central MS SQL `BundleUnlockAttempt` table — the HA store both central nodes already share — rather than in each node's process memory, so the documented cap is a fleet cap and not N × the cap, and a node restart no longer clears it. `BundleUnlockRateLimiter` still owns the policy; `SqlBundleUnlockAttemptWindow` is the store behind the `IBundleUnlockAttemptWindow` seam, and `IBundleUnlockAttemptRepository` (Commons interface, ConfigurationDatabase implementation) is the persistence. Three deliberate trades: **the throttle fails OPEN** — a store error logs a Warning and permits the attempt, because this is a best-effort rate cap over an already-authenticated Administrator-gated surface and the availability of legitimate imports outranks the cap; **accuracy is approximate** — count-then-insert is not atomic across nodes, so simultaneous attempts can overshoot by the number in flight, which is bounded and self-corrects within the window; and **rows are pruned best-effort** (CAS-claimed at most every 15 minutes per process, bounded to 1000 rows, retaining twice the window). That prune originally rode the unlock path alone, since Transport owned no central scheduler — which meant a cluster that stopped importing entirely left its last rows in place indefinitely. **Register row 65 closed that**: Transport's central-only `BundleSessionEvictionService` sweep now also ticks `IBundleUnlockAttemptWindow.PruneAsync`, sharing the one 15-minute cadence claim with the unlock path, so the table converges with no import traffic at all. `PruneAsync` never throws (the fail-open policy stays with the limiter on the attempt path).
- **Bundle size cap** on upload (default 100 MB, configurable) to bound memory.
- **In-transit:** existing HTTPS to the Central UI; no new channel.
- **Audit trail is the chain of custody.** Every export, every import (including aborted ones at validation), and every unlock failure is audit-logged with source env, content hash, encrypted yes/no, and artifact summary.
docs/test_infra/test_infra_db.md
+34
View File
@@ -26,6 +26,40 @@ The test database uses Microsoft SQL Server 2022 Developer Edition running in Do
Both databases are created by `infra/mssql/setup.sql`. EF Core migrations populate the `ScadaBridgeConfig` schema. The `ScadaBridgeMachineData` database is seeded with sample tables and stored procedures by `infra/mssql/machinedata_seed.sql`.
### Ephemeral per-test-class databases
`MsSqlMigrationFixture` (`tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Migrations/`) creates a
throwaway `ScadaBridgeAuditMigTest_<guid>` database per test class, applies the full EF migration chain
to it, and drops it on teardown. Both `ConfigurationDatabase.Tests` and `AuditLog.Tests` use it — the
latter by project reference, so there is one fixture implementation, not a copy. Seeing a handful of
these databases come and go while tests run is normal; seeing them survive a run is not (see below).
Three behaviours are worth knowing when you look at this server during a test run:
- **Concurrent create+migrate is throttled machine-wide.** Fixtures take a slot from a counting
semaphore built on `sp_getapplock` in `master` (resources `ScadaBridgeTestMigrate/0..N-1`) and hold it
across `CREATE DATABASE` + the migration chain. The default is 4 slots; override with
`SCADABRIDGE_MSSQL_TEST_MIGRATE_SLOTS`. An xunit collection attribute cannot do this job — a
solution-level `dotnet test` runs several MSSQL-backed test *processes* at once, so the throttle has
to live on the shared server. Without it, dozens of simultaneous fresh-database migrations saturated
the container and individual DDL statements blew past EF's default 30-second command timeout,
failing whole test classes at fixture construction (deferred-work register row 82).
- **Migration commands use a 300s timeout**, and CREATE/DROP DATABASE use 120s. Under a bounded queue
the server is busy but progressing, so the correct response to a slow statement is to wait.
- **Strays are swept.** Each test process drops, once at start and best-effort, any
`ScadaBridgeAuditMigTest_*` database older than 2 hours, so a run killed before teardown does not
leave databases accumulating. To clear them by hand:
```bash
docker exec -i scadabridge-mssql /opt/mssql-tools18/bin/sqlcmd \
-S localhost -U sa -P 'ScadaBridge_Dev1#' -C \
-Q "SELECT name, create_date FROM sys.databases WHERE name LIKE 'ScadaBridgeAuditMigTest_%'"
```
Overriding the server the fixture targets (CI, a non-default port) is done with
`SCADABRIDGE_MSSQL_TEST_CONN`. When no server answers, the fixture reports `Available = false` and the
tests report as **Skipped**, not passed.
## Data Persistence
SQL data is stored in the named Docker volume `scadabridge-mssql-data`. Data survives container restarts and `docker compose down`. To reset the database completely:
src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogIngestActor.cs
+57 -94
View File
@@ -42,13 +42,14 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Central;
/// placeholder.
/// </para>
/// <para>
/// Two constructors exist for a deliberate reason: the test ctor injects a
/// concrete <see cref="IAuditLogRepository"/> against a per-test MSSQL fixture
/// (the only way to verify the ingest-timestamp stamp + duplicate-key
/// idempotency end to end), while the production host wiring registers the
/// actor as a cluster singleton and must therefore resolve the repository —
/// which is a scoped EF Core service — from a fresh DI scope per message.
/// Mirroring the Notification Outbox actor's pattern.
/// There is exactly ONE constructor, taking an <see cref="IServiceProvider"/>:
/// the actor is a cluster singleton and the repository is a scoped EF Core
/// service, so it must be resolved from a fresh DI scope per message and never
/// held on a field. Mirroring the Notification Outbox actor's pattern. Tests
/// that need a concrete repository (e.g. the MSSQL-backed fixture verifying the
/// ingest-timestamp stamp and duplicate-key idempotency) supply a stub provider
/// that hands one out, rather than a second constructor that would cache it —
/// see register row 61.
/// </para>
/// </remarks>
public class AuditLogIngestActor : ReceiveActor
@@ -100,41 +101,25 @@ public class AuditLogIngestActor : ReceiveActor
/// </remarks>
internal static readonly TimeSpan IngestFallbackBudget = TimeSpan.FromSeconds(5);
private readonly IServiceProvider? _serviceProvider;
private readonly IAuditLogRepository? _injectedRepository;
private readonly IServiceProvider _serviceProvider;
private readonly ILogger<AuditLogIngestActor> _logger;
/// <summary>
/// Test-mode constructor — injects a concrete repository instance whose
/// lifetime exceeds the test, so the actor reuses the same instance across
/// every message. Used by the MSSQL-backed TestKit fixture.
/// </summary>
/// <param name="repository">Audit log repository instance shared across all messages.</param>
/// <param name="logger">Logger for ingest diagnostics.</param>
public AuditLogIngestActor(
IAuditLogRepository repository,
ILogger<AuditLogIngestActor> logger)
{
ArgumentNullException.ThrowIfNull(repository);
ArgumentNullException.ThrowIfNull(logger);
_injectedRepository = repository;
_logger = logger;
ReceiveAsync<IngestAuditEventsCommand>(OnIngestAsync);
// The single-repository test ctor cannot service the dual-write —
// it has no SiteCalls repo and no DbContext. The handler still
// registers (so callers don't dead-letter) but replies empty so the
// test surface stays explicit about what this ctor supports.
ReceiveAsync<IngestCachedTelemetryCommand>(OnCachedTelemetryWithoutDualWriteAsync);
}
/// <summary>
/// Production constructor — resolves <see cref="IAuditLogRepository"/> from
/// The only constructor — resolves <see cref="IAuditLogRepository"/> from
/// a fresh DI scope per message because the repository is a scoped EF Core
/// service registered by <c>AddConfigurationDatabase</c>. The actor itself
/// is a long-lived cluster singleton, so it cannot hold a scope across
/// messages.
///
/// <para>
/// Register row 61: a second, repository-taking "test-mode" constructor used
/// to cache a scoped repository on a field for the actor's whole lifetime.
/// Nothing in production selected it, but it left the captive-dependency bug
/// one call site away and had to be allowlisted in the Host tests'
/// <c>CaptiveDependencyGuard</c>. It is gone; test fixtures build a stub
/// <see cref="IServiceProvider"/> instead, which keeps the actor's own
/// contract "resolve per scope, hold nothing".
/// </para>
/// </summary>
/// <param name="serviceProvider">Root service provider used to open a fresh scope per message.</param>
/// <param name="logger">Logger for ingest diagnostics.</param>
@@ -169,59 +154,51 @@ public class AuditLogIngestActor : ReceiveActor
var accepted = new List<Guid>(cmd.Events.Count);
// Resolve the repository for the whole batch — one DbContext per
// message, mirroring NotificationOutboxActor. The injected-repository
// mode (test ctor) skips the scope entirely.
// message, mirroring NotificationOutboxActor.
// The IAuditRedactor is also resolved from the per-message scope when
// one is available so the row is truncated + redacted before
// InsertIfNotExistsAsync. The single-repository test ctor has no
// service provider — it falls through with no redactor, which preserves
// the small-payload assumptions baked into the existing fixtures.
// InsertIfNotExistsAsync. A composition root that registers no redactor
// (test fixtures) falls back to the SafeDefault inside
// IngestWithRepositoryAsync rather than to pass-through.
// Use CreateAsyncScope + await using so scoped EF Core
// services (IAsyncDisposable DbContexts) dispose asynchronously
// without blocking on sync Dispose() of pending connection cleanup.
if (_injectedRepository is not null)
//
// Guard scope-creation + repository resolution in a
// try/catch, mirroring OnCachedTelemetryAsync. A transient DI /
// DbContext-factory fault (pooled-context init, SQL-connection
// exhaustion, a resolution race during host churn) would otherwise
// propagate out of the ReceiveAsync handler, trip the parent's
// supervision, and RESTART this central singleton over a transient
// fault — dropping the captured reply so the site's Ask times out.
// Best-effort audit must never wedge the singleton: log, optionally
// bump the failure counter, and still reply with whatever was
// accepted (empty on an up-front scope-resolution throw) so the
// site keeps its rows Pending and retries on the next drain.
try
{
await IngestWithRepositoryAsync(_injectedRepository, redactor: null, failureCounter: null, cmd, nowUtc, accepted)
await using var scope = _serviceProvider.CreateAsyncScope();
var repository = scope.ServiceProvider.GetRequiredService<IAuditLogRepository>();
var redactor = scope.ServiceProvider.GetService<IAuditRedactor>();
// Central health counter is best-effort —
// unregistered (test composition roots) means the per-row catch
// simply logs without surfacing on the health dashboard.
var failureCounter = scope.ServiceProvider.GetService<ICentralAuditWriteFailureCounter>();
await IngestWithRepositoryAsync(repository, redactor, failureCounter, cmd, nowUtc, accepted)
.ConfigureAwait(false);
}
else
catch (Exception ex)
{
// Guard scope-creation + repository resolution in a
// try/catch, mirroring OnCachedTelemetryAsync. A transient DI /
// DbContext-factory fault (pooled-context init, SQL-connection
// exhaustion, a resolution race during host churn) would otherwise
// propagate out of the ReceiveAsync handler, trip the parent's
// supervision, and RESTART this central singleton over a transient
// fault — dropping the captured reply so the site's Ask times out.
// Best-effort audit must never wedge the singleton: log, optionally
// bump the failure counter, and still reply with whatever was
// accepted (empty on an up-front scope-resolution throw) so the
// site keeps its rows Pending and retries on the next drain.
try
{
await using var scope = _serviceProvider!.CreateAsyncScope();
var repository = scope.ServiceProvider.GetRequiredService<IAuditLogRepository>();
var redactor = scope.ServiceProvider.GetService<IAuditRedactor>();
// Central health counter is best-effort —
// unregistered (test composition roots) means the per-row catch
// simply logs without surfacing on the health dashboard.
var failureCounter = scope.ServiceProvider.GetService<ICentralAuditWriteFailureCounter>();
await IngestWithRepositoryAsync(repository, redactor, failureCounter, cmd, nowUtc, accepted)
.ConfigureAwait(false);
}
catch (Exception ex)
{
// Scope creation or a required-service resolution threw before
// (or while) processing the batch. Surface a sustained fault on
// the dashboard if the counter is registered, but never let the
// throw escape the handler and restart the singleton.
try { _serviceProvider!.GetService<ICentralAuditWriteFailureCounter>()?.Increment(); }
catch { /* counter must never throw — defence in depth */ }
_logger.LogError(
ex,
"Audit event batch ingest failed before/while resolving the repository scope; replying with {Accepted} accepted row(s). The site keeps unaccepted rows Pending and retries on the next drain.",
accepted.Count);
}
// Scope creation or a required-service resolution threw before
// (or while) processing the batch. Surface a sustained fault on
// the dashboard if the counter is registered, but never let the
// throw escape the handler and restart the singleton.
try { _serviceProvider.GetService<ICentralAuditWriteFailureCounter>()?.Increment(); }
catch { /* counter must never throw — defence in depth */ }
_logger.LogError(
ex,
"Audit event batch ingest failed before/while resolving the repository scope; replying with {Accepted} accepted row(s). The site keeps unaccepted rows Pending and retries on the next drain.",
accepted.Count);
}
replyTo.Tell(new IngestAuditEventsReply(accepted));
@@ -357,7 +334,7 @@ public class AuditLogIngestActor : ReceiveActor
try
{
await using var scope = _serviceProvider!.CreateAsyncScope();
await using var scope = _serviceProvider.CreateAsyncScope();
var auditRepo = scope.ServiceProvider.GetRequiredService<IAuditLogRepository>();
var siteCallRepo = scope.ServiceProvider.GetRequiredService<ISiteCallAuditRepository>();
var dbContext = scope.ServiceProvider.GetRequiredService<ScadaBridgeDbContext>();
@@ -544,18 +521,4 @@ public class AuditLogIngestActor : ReceiveActor
return true;
}
/// <summary>
/// Fallback handler installed on the single-repository test ctor — that
/// ctor has no DbContext and no <see cref="ISiteCallAuditRepository"/>, so
/// it cannot service the dual-write. Logs a warning and replies with an
/// empty ack so callers fall through to their retry path.
/// </summary>
private Task OnCachedTelemetryWithoutDualWriteAsync(IngestCachedTelemetryCommand cmd)
{
_logger.LogWarning(
"AuditLogIngestActor received IngestCachedTelemetryCommand on the single-repository ctor; dual-write requires the IServiceProvider ctor. Replying with empty ack ({Count} entries).",
cmd.Entries.Count);
Sender.Tell(new IngestCachedTelemetryReply(Array.Empty<Guid>()));
return Task.CompletedTask;
}
}
src/ZB.MOM.WW.ScadaBridge.AuditLog/ServiceCollectionExtensions.cs
+32 -9
View File
@@ -248,15 +248,38 @@ public static class ServiceCollectionExtensions
// provider as a singleton on both site and central paths.
sp.GetRequiredService<INodeIdentityProvider>()));
// KPI History & Trends: the AuditLog source the central KPI
// recorder enumerates each sampling pass to snapshot Global-scope Audit
// Log KPIs (volume / error rate / backlog) into the KpiSample history
// table. Scoped to match its IAuditLogRepository dependency — a SCOPED
// EF Core service; the recorder opens a fresh scope per sampling pass.
// TryAdd-Enumerable so multiple sources can register the same interface
// (NotificationOutbox + SiteCallAudit + SiteHealth add their own) without
// any one composition root clobbering another, and re-registration is a
// no-op if AddAuditLog were ever called twice.
return services;
}
/// <summary>
/// Registers the Audit Log's KPI sample source. CENTRAL-ONLY — call from the Central
/// composition root only, alongside <c>AddKpiHistory</c>.
/// </summary>
/// <remarks>
/// <para>
/// KPI History &amp; Trends: the AuditLog source the central KPI recorder enumerates each
/// sampling pass to snapshot Global-scope Audit Log KPIs (volume / error rate / backlog)
/// into the <c>KpiSample</c> history table. Scoped to match its <c>IAuditLogRepository</c>
/// dependency — a SCOPED EF Core service; the recorder opens a fresh scope per sampling
/// pass. TryAdd-Enumerable so multiple sources can register the same interface
/// (NotificationOutbox + SiteCallAudit + SiteHealth add their own) without any one
/// composition root clobbering another, and re-registration is a no-op if this were ever
/// called twice.
/// </para>
/// <para>
/// Split out of <see cref="AddAuditLog"/> by deferred register row 50. Both the KPI
/// recorder and <c>IAuditLogRepository</c> are central-only, so on a site node this
/// registration was unconstructible — enough on its own to fail <c>ValidateOnBuild</c> for
/// the entire site graph and so to keep scope validation switched off there. The site half
/// of <see cref="AddAuditLog"/> (the SQLite writer chain) is unaffected and stays shared.
/// </para>
/// </remarks>
/// <param name="services">The DI service collection to register services into.</param>
/// <returns>The same <see cref="IServiceCollection"/> to allow chaining.</returns>
public static IServiceCollection AddAuditLogCentralKpi(this IServiceCollection services)
{
ArgumentNullException.ThrowIfNull(services);
services.TryAddEnumerable(
ServiceDescriptor.Scoped<IKpiSampleSource, AuditLogKpiSampleSource>());
src/ZB.MOM.WW.ScadaBridge.CLI/Commands/AuditConfigTableProjection.cs
+153
View File
@@ -0,0 +1,153 @@
using System.Text.Json;
using System.Text.Json.Nodes;
namespace ZB.MOM.WW.ScadaBridge.CLI.Commands;
/// <summary>
/// Compact table projection for <c>audit-config query</c> (deferred-work register row 57).
///
/// <para>
/// The handler returns an <c>AuditLogQueryResult</c> — <c>{ "entries": [ … ],
/// "totalCount": N }</c>. Handed to the generic renderer unprojected, that object takes
/// the Property/Value branch and the whole <c>entries</c> array lands in a single cell as
/// raw JSON, so <c>--format table</c> was unreadable. This unwraps the envelope to the
/// row array the renderer expects and drops <c>afterStateJson</c>, the per-entry snapshot
/// of the entire changed entity — the same "one fat field makes the table useless" problem
/// <see cref="TemplateTableProjection"/> solves by dropping child collections.
/// </para>
///
/// <para>
/// <c>totalCount</c> would be lost by a plain unwrap, so when the page is a strict subset
/// of the matches a single trailing row carries a <c>note</c> cell saying how many of how
/// many are shown. Every real entry leaves that cell empty, and when the page IS the whole
/// result set the column is not emitted at all — so the common small-query table stays
/// exactly as narrow as its data.
/// </para>
///
/// <para>
/// JSON output is never touched (this runs only on the table path), and the command's
/// <c>--detail</c> flag skips the projection to render the raw payload as-is — that is
/// the way to see <c>afterStateJson</c> in table form.
/// </para>
/// </summary>
internal static class AuditConfigTableProjection
{
/// <summary>
/// Projects an <c>AuditLogQueryResult</c> envelope (or a bare entry array) to the
/// compact row array the table renderer consumes. Returns the input unchanged when it
/// is not JSON or not one of those shapes, so the renderer's own fallbacks still apply.
/// </summary>
/// <param name="json">The raw success JSON body from the management API.</param>
/// <returns>Compact JSON suitable for table rendering.</returns>
internal static string ProjectEntries(string json)
{
JsonDocument doc;
try
{
doc = JsonDocument.Parse(json);
}
catch (JsonException)
{
// Not JSON (e.g. a proxy error page) — let the renderer print it verbatim.
return json;
}
using (doc)
{
var root = doc.RootElement;
// A bare array is accepted so the projection still works if the payload is
// ever unwrapped upstream, and so the deprecated `audit-log` alias against an
// older server does not render worse than it does today.
if (root.ValueKind == JsonValueKind.Array)
{
return BuildRows(root, totalCount: null).ToJsonString();
}
if (root.ValueKind == JsonValueKind.Object
&& TryGetPropertyCI(root, "entries", out var entries)
&& entries.ValueKind == JsonValueKind.Array)
{
int? total = TryGetPropertyCI(root, "totalCount", out var tc)
&& tc.ValueKind == JsonValueKind.Number
&& tc.TryGetInt32(out var n)
? n
: null;
return BuildRows(entries, total).ToJsonString();
}
// Some other object (e.g. an error envelope) — leave it to the renderer.
return json;
}
}
private static JsonArray BuildRows(JsonElement entries, int? totalCount)
{
var arr = new JsonArray();
var shown = 0;
foreach (var item in entries.EnumerateArray())
{
arr.Add(ProjectElement(item));
shown++;
}
// Only when the page is a strict subset — otherwise the count is just the row
// count the operator can already see, and the extra column would be noise.
if (totalCount is int total && total > shown)
{
arr.Add(new JsonObject
{
["note"] = JsonValue.Create(
$"showing {shown} of {total} — use --page / --page-size for the rest"),
});
}
return arr;
}
/// <summary>Projects a single audit entry to its compact row node.</summary>
private static JsonNode ProjectElement(JsonElement element)
{
if (element.ValueKind != JsonValueKind.Object)
return JsonValue.Create(element.ToString())!;
// JsonObject preserves insertion order, which fixes the table's column order.
// afterStateJson (the full post-change entity snapshot) and bundleImportId are
// deliberately omitted — `--detail`, or --format json, for those.
return new JsonObject
{
["id"] = Int(element, "id"),
["timestamp"] = Str(element, "timestamp"),
["user"] = Str(element, "user"),
["action"] = Str(element, "action"),
["entityType"] = Str(element, "entityType"),
["entityName"] = Str(element, "entityName"),
["entityId"] = Str(element, "entityId"),
};
}
private static bool TryGetPropertyCI(JsonElement obj, string name, out JsonElement value)
{
foreach (var prop in obj.EnumerateObject())
{
if (string.Equals(prop.Name, name, StringComparison.OrdinalIgnoreCase))
{
value = prop.Value;
return true;
}
}
value = default;
return false;
}
private static JsonNode? Str(JsonElement obj, string name)
=> TryGetPropertyCI(obj, name, out var v) && v.ValueKind == JsonValueKind.String
? JsonValue.Create(v.GetString())
: null;
private static JsonNode? Int(JsonElement obj, string name)
=> TryGetPropertyCI(obj, name, out var v) && v.ValueKind == JsonValueKind.Number && v.TryGetInt32(out var n)
? JsonValue.Create(n)
: null;
}
src/ZB.MOM.WW.ScadaBridge.CLI/Commands/AuditLogCommands.cs
+11 -1
View File
@@ -75,6 +75,13 @@ public static class AuditLogCommands
pageOption.DefaultValueFactory = _ => 1;
var pageSizeOption = new Option<int>("--page-size") { Description = "Page size" };
pageSizeOption.DefaultValueFactory = _ => 50;
var detailOption = new Option<bool>("--detail")
{
Description = "Render the raw query payload in table output instead of the compact column projection. "
+ "This is how to see 'afterStateJson' (the full post-change entity snapshot) in table form — it is "
+ "omitted from the compact columns because a single entry's snapshot fills the terminal. "
+ "No effect on JSON output."
};
var cmd = new Command("query") { Description = "Query audit log entries" };
cmd.Add(userOption);
@@ -84,6 +91,7 @@ public static class AuditLogCommands
cmd.Add(toOption);
cmd.Add(pageOption);
cmd.Add(pageSizeOption);
cmd.Add(detailOption);
cmd.SetAction(async (ParseResult result) =>
{
var user = result.GetValue(userOption);
@@ -93,9 +101,11 @@ public static class AuditLogCommands
var to = result.GetValue(toOption);
var page = result.GetValue(pageOption);
var pageSize = result.GetValue(pageSizeOption);
var detail = result.GetValue(detailOption);
return await CommandHelpers.ExecuteCommandAsync(
result, urlOption, formatOption, usernameOption, passwordOption,
new QueryAuditLogCommand(user, entityType, action, from, to, page, pageSize));
new QueryAuditLogCommand(user, entityType, action, from, to, page, pageSize),
tableProjector: detail ? null : AuditConfigTableProjection.ProjectEntries);
});
return cmd;
}
src/ZB.MOM.WW.ScadaBridge.CLI/Commands/ExternalSystemCommands.cs
+20 -2
View File
@@ -50,6 +50,9 @@ public static class ExternalSystemCommands
var authTypeOption = new Option<string>("--auth-type") { Description = "Auth type", Required = true };
var authConfigOption = new Option<string?>("--auth-config") { Description = "Auth configuration JSON" };
var timeoutOption = new Option<int?>("--timeout-seconds") { Description = "Per-system HTTP timeout in seconds (0 = use gateway default); omit to preserve" };
var maxRetriesOption = new Option<int?>("--max-retries") { Description = "Store-and-forward retry budget for cached calls (0 = use engine default); omit to preserve" };
var retryDelayOption = new Option<int?>("--retry-delay-seconds") { Description = "Fixed interval between store-and-forward retries, in seconds (0 = use engine default); omit to preserve" };
retryDelayOption.Aliases.Add("--retry-delay");
var cmd = new Command("update") { Description = "Update an external system" };
cmd.Add(idOption);
@@ -58,6 +61,8 @@ public static class ExternalSystemCommands
cmd.Add(authTypeOption);
cmd.Add(authConfigOption);
cmd.Add(timeoutOption);
cmd.Add(maxRetriesOption);
cmd.Add(retryDelayOption);
cmd.SetAction(async (ParseResult result) =>
{
var id = result.GetValue(idOption);
@@ -66,9 +71,13 @@ public static class ExternalSystemCommands
var authType = result.GetValue(authTypeOption)!;
var authConfig = result.GetValue(authConfigOption);
var timeoutSeconds = result.GetValue(timeoutOption);
var maxRetries = result.GetValue(maxRetriesOption);
var retryDelaySeconds = result.GetValue(retryDelayOption);
return await CommandHelpers.ExecuteCommandAsync(
result, urlOption, formatOption, usernameOption, passwordOption,
new UpdateExternalSystemCommand(id, name, endpointUrl, authType, authConfig, timeoutSeconds));
new UpdateExternalSystemCommand(id, name, endpointUrl, authType, authConfig, timeoutSeconds,
maxRetries,
retryDelaySeconds is { } rd ? TimeSpan.FromSeconds(rd) : null));
});
return cmd;
}
@@ -91,6 +100,9 @@ public static class ExternalSystemCommands
var authTypeOption = new Option<string>("--auth-type") { Description = "Auth type (ApiKey, BasicAuth)", Required = true };
var authConfigOption = new Option<string?>("--auth-config") { Description = "Auth configuration JSON" };
var timeoutOption = new Option<int?>("--timeout-seconds") { Description = "Per-system HTTP timeout in seconds (0 = use gateway default)" };
var maxRetriesOption = new Option<int?>("--max-retries") { Description = "Store-and-forward retry budget for cached calls (0 = use engine default)" };
var retryDelayOption = new Option<int?>("--retry-delay-seconds") { Description = "Fixed interval between store-and-forward retries, in seconds (0 = use engine default)" };
retryDelayOption.Aliases.Add("--retry-delay");
var cmd = new Command("create") { Description = "Create an external system" };
cmd.Add(nameOption);
@@ -98,6 +110,8 @@ public static class ExternalSystemCommands
cmd.Add(authTypeOption);
cmd.Add(authConfigOption);
cmd.Add(timeoutOption);
cmd.Add(maxRetriesOption);
cmd.Add(retryDelayOption);
cmd.SetAction(async (ParseResult result) =>
{
var name = result.GetValue(nameOption)!;
@@ -105,9 +119,13 @@ public static class ExternalSystemCommands
var authType = result.GetValue(authTypeOption)!;
var authConfig = result.GetValue(authConfigOption);
var timeoutSeconds = result.GetValue(timeoutOption);
var maxRetries = result.GetValue(maxRetriesOption);
var retryDelaySeconds = result.GetValue(retryDelayOption);
return await CommandHelpers.ExecuteCommandAsync(
result, urlOption, formatOption, usernameOption, passwordOption,
new CreateExternalSystemCommand(name, endpointUrl, authType, authConfig, timeoutSeconds));
new CreateExternalSystemCommand(name, endpointUrl, authType, authConfig, timeoutSeconds,
maxRetries,
retryDelaySeconds is { } rd ? TimeSpan.FromSeconds(rd) : null));
});
return cmd;
}
src/ZB.MOM.WW.ScadaBridge.CLI/README.md
+77 -3
View File
@@ -1206,7 +1206,7 @@ scadabridge --url <url> external-system list
Register an external HTTP system that scripts can call.
```sh
scadabridge --url <url> external-system create --name <string> --endpoint-url <url> --auth-type <string> [--auth-config <json>]
scadabridge --url <url> external-system create --name <string> --endpoint-url <url> --auth-type <string> [--auth-config <json>] [--timeout-seconds <int>] [--max-retries <int>] [--retry-delay-seconds <int>]
```
| Option | Required | Description |
@@ -1215,6 +1215,13 @@ scadabridge --url <url> external-system create --name <string> --endpoint-url <u
| `--endpoint-url` | yes | Base URL of the external system |
| `--auth-type` | yes | Authentication type: `ApiKey` or `BasicAuth` |
| `--auth-config` | no | Auth credentials as a JSON string |
| `--timeout-seconds` | no | Per-system HTTP round-trip timeout. `0` (default) = use the gateway's `DefaultHttpTimeout` |
| `--max-retries` | no | Store-and-forward retry budget for this system's `CachedCall`s. `0` (default) = use the engine's `DefaultMaxRetries` (50) |
| `--retry-delay-seconds` | no | Fixed interval between store-and-forward retries. `0` (default) = use the engine's `DefaultRetryInterval` (30 s). Alias: `--retry-delay` |
The retry options are deployed to sites with the rest of the definition, so they take
effect on the next deployment — they govern how long a transient outage of this system is
buffered before a cached call parks (`--max-retries` x `--retry-delay-seconds`).
#### `external-system update`
@@ -1222,7 +1229,7 @@ Update an external system definition. An update **replaces** the whole entity
required field below must be supplied, even if unchanged.
```sh
scadabridge --url <url> external-system update --id <int> --name <string> --endpoint-url <url> --auth-type <string> [--auth-config <json>]
scadabridge --url <url> external-system update --id <int> --name <string> --endpoint-url <url> --auth-type <string> [--auth-config <json>] [--timeout-seconds <int>] [--max-retries <int>] [--retry-delay-seconds <int>]
```
| Option | Required | Description |
@@ -1231,7 +1238,14 @@ scadabridge --url <url> external-system update --id <int> --name <string> --endp
| `--name` | yes | Display name |
| `--endpoint-url` | yes | Base URL |
| `--auth-type` | yes | Authentication type |
| `--auth-config` | no | Updated auth credentials as a JSON string |
| `--auth-config` | no | Updated auth credentials as a JSON string; omit to preserve the stored value |
| `--timeout-seconds` | no | Per-system HTTP round-trip timeout; omit to preserve |
| `--max-retries` | no | Store-and-forward retry budget for this system's `CachedCall`s; omit to preserve |
| `--retry-delay-seconds` | no | Fixed interval between store-and-forward retries; omit to preserve. Alias: `--retry-delay` |
Note the asymmetry with the required fields above: the four optional settings follow an
**omit-to-preserve** rule, so a rename does not silently reset a system's auth config,
timeout, or retry policy.
#### `external-system delete`
@@ -1801,6 +1815,66 @@ scadabridge --url <url> audit-config query [options]
| `--to` | no | — | End timestamp in ISO 8601 format |
| `--page` | no | `1` | Page number |
| `--page-size` | no | `50` | Results per page |
| `--detail` | no | `false` | Table output only: render the raw payload instead of the compact column projection (see below). No effect on JSON output. |
Output shape:
```json
{
"entries": [
{
"id": 412,
"user": "multi-role",
"action": "Created",
"entityType": "Template",
"entityId": "7",
"entityName": "PumpTemplate",
"afterStateJson": "{\"Id\":7,\"Name\":\"PumpTemplate\",\"FolderId\":3}",
"timestamp": "2026-08-15T12:00:00+00:00",
"bundleImportId": null
},
{
"id": 413,
"user": "designer",
"action": "Deleted",
"entityType": "ExternalSystem",
"entityId": "3",
"entityName": "MES",
"afterStateJson": null,
"timestamp": "2026-08-15T12:05:00+00:00",
"bundleImportId": "9f2c1e40-5b31-4a7e-8d66-0c1a4b93ff21"
}
],
"totalCount": 137
}
```
Both nullable fields are always present on the wire. `afterStateJson` is a JSON snapshot
of the entity's state *after* the change, embedded as a string — it is `null` for deletes,
since there is no after-state. `bundleImportId` is the import session id when the change
was made by a bundle import (`transport import`), and `null` for a change made directly;
it is the join key for "what did this bundle actually change".
`totalCount` is the unpaged number of rows matching the filters, so a client can page
without a second call. JSON is the primary format for this command and is what scripts
should consume — it is never altered by the table projection below.
**Table output.** `--format table` renders a compact projection with one row per entry:
```
id timestamp user action entityType entityName entityId
412 2026-08-15T12:00:00+00:00 multi-role Created Template PumpTemplate 7
413 2026-08-15T12:05:00+00:00 designer Deleted ExternalSystem MES 3
```
`afterStateJson` (the full post-change entity snapshot) and `bundleImportId` are omitted
— one snapshot per row fills a terminal. Pass `--detail` to render the raw payload
instead, or use `--format json`. When the page is a strict subset of the matches, a final
row carries a `note` cell (`showing 50 of 137 — use --page / --page-size for the rest`);
when the page is the whole result set that column is not emitted at all.
Before this projection existed the whole `entries` array was printed as raw JSON inside a
single `Property`/`Value` cell (deferred-work register row 57).
---
src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Health/CentralFailoverControl.razor
+4 -2
View File
@@ -70,14 +70,16 @@
+ "standby takes over and its Deployment Manager singleton hands over gracefully. "
+ "In-flight work on that site node is interrupted, and the site is briefly "
+ "unavailable while the handover completes.",
danger: true)
danger: true,
confirmLabel: "Fail over")
: await Dialog.ConfirmAsync(
"Trigger central failover?",
"The active central node will leave the cluster and restart; the standby takes over "
+ "and becomes active. Cluster singletons hand over gracefully, but in-flight work on "
+ "the active node is interrupted. This page is served by the active node, so it will "
+ "briefly disconnect and reconnect against the new active node.",
danger: true);
danger: true,
confirmLabel: "Fail over");
if (!confirmed)
{
src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Deployment/Topology.razor
+2 -1
View File
@@ -868,7 +868,8 @@
var confirmed = await Dialog.ConfirmAsync(
"Disable Instance",
$"Disable instance '{inst.UniqueName}'? The instance actor will be stopped.",
danger: true);
danger: true,
confirmLabel: "Disable");
if (!confirmed) return;
_actionInProgress = true;
src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/Health.razor
+69
View File
@@ -170,6 +170,17 @@
IsAvailable="@_sfBufferDepthAvailable"
ErrorMessage="@_sfBufferDepthError" />
</div>
@* Register row 33. The point of charting the backlog is retrospective:
the daily purge burst is expected, so what an operator needs is to see
whether a past spike DRAINED — which a point-in-time tile can never
answer. An unreplicated site emits no samples, so this chart is
legitimately empty there rather than a flat, misleading zero line. *@
<div class="col-lg-3 col-md-6">
<KpiTrendChart Title="LocalDb Replication Backlog"
Points="@_localDbBacklogSeries"
IsAvailable="@_localDbBacklogAvailable"
ErrorMessage="@_localDbBacklogError" />
</div>
</div>
}
</div>
@@ -248,6 +259,22 @@
Metrics stale
</span>
}
@if (IsLocalDbBacklogSustained(state))
{
<span class="badge bg-warning text-dark"
data-test="localdb-backlog-badge"
title="LocalDb replication oplog backlog has stayed above @HealthOptions.Value.LocalDbBacklogThreshold for over @BacklogSustainedDisplay. A short daily spike at the site_events purge is expected and is NOT this — the peer has stopped draining.">
Replication backlog
</span>
}
@if (IsStartupReconcilerMissing(state))
{
<span class="badge bg-warning text-dark"
data-test="startup-reconciler-badge"
title="This site reported that a node booted without its SiteReconciliationActor (no IDeploymentConfigFetcher), so the startup self-heal is disabled: a deploy that lands while the node is DOWN will not be recovered on restart. Config still reaches the node via LocalDb replication while it is up. Check the node's configuration.">
No self-heal
</span>
}
@if (!state.IsOnline && state.LastStatusChangeAt is { } changedAt)
{
<small class="text-muted">offline since @changedAt.ToString("u")</small>
@@ -526,6 +553,43 @@
private string StaleTimeoutDisplay =>
FormatDuration(HealthOptions.Value.MetricsStaleTimeout);
// Human-friendly rendering of the configured sustained-backlog dwell for the
// "Replication backlog" badge tooltip (e.g. "15 minutes").
private string BacklogSustainedDisplay =>
FormatDuration(HealthOptions.Value.LocalDbBacklogSustainedFor);
// Register row 33: the sustained LocalDb replication backlog badge.
//
// Reads the anchor the aggregator maintains (LocalDbBacklogAboveSince) rather than the
// instantaneous backlog on the report, because a bare "backlog > threshold" badge would
// fire on EVERY replicated site EVERY day: the site_events retention purge is CDC-captured
// like any other write, so the backlog jumps by the deleted batch size and then drains.
// Healthy is "the backlog comes back down"; only a dwell can express that.
//
// A site that reports no backlog at all (unreplicated — rig site-b/site-c — or a failed
// poll) never gets an anchor, so it can never be badged. That is the whole null-semantics
// requirement: null means "no reading", not zero.
private bool IsLocalDbBacklogSustained(SiteHealthState state) =>
state.IsOnline
&& state.LocalDbBacklogAboveSince is { } since
&& DateTimeOffset.UtcNow - since >= HealthOptions.Value.LocalDbBacklogSustainedFor;
// Register row 73: the "No self-heal" badge.
//
// Warning-level treatment, deliberately not an error and never an offline-style state:
// the node still receives config through LocalDb replication while it is up, so what it
// has lost is recovery from a deploy that landed while it was DOWN. That is degraded,
// not down — which is exactly what a warning badge says.
//
// Tri-state, and only the explicit false is badged. Null means "not reported": the
// central self-report row (where a site reconciler is meaningless) and any site node on
// a pre-row-73 build both send null, and badging those would fire on every site the
// moment central is upgraded ahead of the sites. Offline sites are skipped for the same
// reason the backlog badge skips them — the Offline badge is the signal that matters.
private bool IsStartupReconcilerMissing(SiteHealthState state) =>
state.IsOnline
&& state.LatestReport?.StartupReconcilerActive == false;
// Online central nodes, from the same ClusterNodes list the Nodes column renders.
// Fewer than two means no standby, which disables the manual-failover control. This is
// a display-side guard only — AkkaManualFailoverService re-checks against live cluster
@@ -616,6 +680,9 @@
private IReadOnlyList<KpiSeriesPoint>? _sfBufferDepthSeries;
private bool _sfBufferDepthAvailable = true;
private string? _sfBufferDepthError;
private IReadOnlyList<KpiSeriesPoint>? _localDbBacklogSeries;
private bool _localDbBacklogAvailable = true;
private string? _localDbBacklogError;
private static bool SiteHasActiveErrors(SiteHealthState state)
{
@@ -739,6 +806,8 @@
await LoadTrendSeriesAsync(KpiMetrics.SiteHealth.ScriptErrors, siteId, fromUtc, toUtc);
(_sfBufferDepthSeries, _sfBufferDepthAvailable, _sfBufferDepthError) =
await LoadTrendSeriesAsync(KpiMetrics.SiteHealth.SfBufferDepth, siteId, fromUtc, toUtc);
(_localDbBacklogSeries, _localDbBacklogAvailable, _localDbBacklogError) =
await LoadTrendSeriesAsync(KpiMetrics.SiteHealth.LocalDbOplogBacklog, siteId, fromUtc, toUtc);
}
finally
{
src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/ParkedMessages.razor
+4 -2
View File
@@ -668,7 +668,8 @@
var confirmed = await Dialog.ConfirmAsync(
"Discard parked messages",
$"Permanently discard {ids.Count} message{(ids.Count == 1 ? "" : "s")}? This cannot be undone.",
danger: true);
danger: true,
confirmLabel: "Discard");
if (!confirmed) return;
_bulkInProgress = true;
@@ -720,7 +721,8 @@
var confirmed = await Dialog.ConfirmAsync(
"Discard parked message",
$"Permanently discard message {ShortId(msg.MessageId)}? This cannot be undone.",
danger: true);
danger: true,
confirmLabel: "Discard");
if (!confirmed) return false;
_actionInProgress = true;
src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Notifications/NotificationReport.razor
+2 -1
View File
@@ -566,7 +566,8 @@
var confirmed = await Dialog.ConfirmAsync(
"Discard notification",
$"Permanently discard notification {ShortId(n.NotificationId)} (\"{n.Subject}\")? This cannot be undone.",
danger: true);
danger: true,
confirmLabel: "Discard");
if (!confirmed) return;
_actionInProgress = true;
src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Operations/SecuredWrites.razor
+16 -2
View File
@@ -424,7 +424,14 @@
$"Value: {row.ValueJson} ({row.ValueType})\n" +
$"Submitted {FormatAge(row.SubmittedAtUtc)} ago ({row.SubmittedAtUtc:u})\n" +
$"Operator: {row.OperatorUser}",
danger: true);
// Caution, NOT danger (deferred-work register row 75). Approving executes a
// real write to live plant equipment, so this is not a benign primary action
// — but it is not destroying anything either, and red made it look like a
// delete while rendering it identical to the Reject button right beside it.
// Amber keeps the beat of hesitation the write deserves and restores the
// colour contrast between the two opposite decisions.
DialogSeverity.Caution,
confirmLabel: "Approve");
if (!confirmed) return;
_actionInProgress = true;
@@ -460,7 +467,14 @@
var confirmed = await Dialog.ConfirmAsync(
"Reject secured write",
$"Reject the secured write to {row.SiteId}/{row.ConnectionName}/{row.TagPath}?",
danger: true);
// Reject KEEPS danger styling (register row 75). It is terminal and
// irreversible for the request — the Operator has to resubmit — and holding
// it red is what makes the pair readable: with Approve amber, the two
// opposite decisions no longer look alike. Dropping Reject to a neutral
// button would just invert the bias, making declining look like the safe
// default.
danger: true,
confirmLabel: "Reject");
if (!confirmed) return;
_actionInProgress = true;
src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/SiteCalls/SiteCallsReport.razor.cs
+2 -1
View File
@@ -343,7 +343,8 @@ public partial class SiteCallsReport
"Discard cached call",
$"Relay a discard of cached call {ShortId(c.TrackedOperationId)} (\"{c.Target}\") " +
$"to site {SiteName(c.SourceSite)}? This cannot be undone.",
danger: true);
danger: true,
confirmLabel: "Discard");
if (!confirmed) return;
_actionInProgress = true;
src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Shared/DialogHost.razor
+27 -4
View File
@@ -50,7 +50,7 @@
<div class="btn-group btn-group-sm" role="group">
<button type="button" class="btn btn-outline-secondary" @onclick="Cancel">Cancel</button>
<button type="button"
class="btn @(state.Danger ? "btn-danger" : "btn-primary")"
class="btn @ConfirmButtonClass(state)"
@onclick="Confirm">
@ConfirmLabel(state)
</button>
@@ -185,9 +185,32 @@
svc.Resolve(resolveValue);
}
private static string ConfirmLabel(DialogState state) => state.Kind switch
// Bootstrap class for the confirm button, by severity (deferred-work register row 75).
// Caution is the middle rung: amber, for an action that is consequential but is not
// destroying anything — the Secured Writes "Approve" is the case it was added for,
// where red made the affirmative half of a two-person control look like a delete and
// made Approve and Reject visually indistinguishable. Colour is never the only signal:
// the button is always labelled with its verb.
private static string ConfirmButtonClass(DialogState state) => state.Severity switch
{
DialogKind.Prompt => "Save",
_ => state.Danger ? "Delete" : "Confirm",
DialogSeverity.Destructive => "btn-danger",
DialogSeverity.Caution => "btn-warning",
_ => "btn-primary",
};
// An explicit label from the caller always wins. The kind/severity fallback below
// is a convenience for the common cases only: a bare destructive confirm is assumed
// to be a delete, which is exactly the assumption that put "Delete" on the Trigger
// failover button (deferred-work register row 28). Callers whose destructive verb
// is not "Delete" must pass ConfirmLabel — see IDialogService.ConfirmAsync.
// Note Caution deliberately does NOT inherit that fallback: an unlabelled caution
// dialog reads "Confirm", because a caution action is by definition not a delete.
private static string ConfirmLabel(DialogState state) =>
!string.IsNullOrWhiteSpace(state.ConfirmLabel)
? state.ConfirmLabel
: state.Kind switch
{
DialogKind.Prompt => "Save",
_ => state.Severity == DialogSeverity.Destructive ? "Delete" : "Confirm",
};
}
src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Shared/DialogService.cs
+64 -7
View File
@@ -35,12 +35,27 @@ public class DialogService : IDialogService
private TaskCompletionSource<object?>? _tcs;
/// <inheritdoc />
public Task<bool> ConfirmAsync(string title, string message, bool danger = false)
public Task<bool> ConfirmAsync(string title, string message, bool danger = false, string? confirmLabel = null) =>
ConfirmAsync(
title,
message,
danger ? DialogSeverity.Destructive : DialogSeverity.Normal,
confirmLabel);
/// <inheritdoc />
public Task<bool> ConfirmAsync(string title, string message, DialogSeverity severity, string? confirmLabel = null)
{
EnsureNoActiveDialog();
var tcs = new TaskCompletionSource<object?>(TaskCreationOptions.RunContinuationsAsynchronously);
_tcs = tcs;
Current = new DialogState(title, DialogKind.Confirm, message, danger, PromptInitial: string.Empty, Placeholder: null);
Current = new DialogState(
title,
DialogKind.Confirm,
message,
severity,
PromptInitial: string.Empty,
Placeholder: null,
ConfirmLabel: confirmLabel);
OnChange?.Invoke();
return Project(tcs.Task, static r => r is bool b && b);
}
@@ -51,7 +66,7 @@ public class DialogService : IDialogService
EnsureNoActiveDialog();
var tcs = new TaskCompletionSource<object?>(TaskCreationOptions.RunContinuationsAsynchronously);
_tcs = tcs;
Current = new DialogState(title, DialogKind.Prompt, label, Danger: false, PromptInitial: initialValue, Placeholder: placeholder);
Current = new DialogState(title, DialogKind.Prompt, label, DialogSeverity.Normal, PromptInitial: initialValue, Placeholder: placeholder);
OnChange?.Invoke();
return Project(tcs.Task, static r => r as string);
}
@@ -78,7 +93,7 @@ public class DialogService : IDialogService
title,
DialogKind.Custom,
Body: string.Empty,
Danger: false,
DialogSeverity.Normal,
PromptInitial: string.Empty,
Placeholder: null,
Content: content,
@@ -139,22 +154,28 @@ public class DialogService : IDialogService
/// <param name="Title">Modal title text.</param>
/// <param name="Kind">Discriminates between confirm, prompt, and custom rendering.</param>
/// <param name="Body">For confirm: the message; for prompt: the input label; ignored for custom.</param>
/// <param name="Danger">When true, the confirm button uses danger styling.</param>
/// <param name="Severity">Drives the confirm button's styling, and the verb the host
/// falls back to when <paramref name="ConfirmLabel"/> is blank. Single source of truth
/// for both — there is deliberately no separate boolean, so the two cannot disagree.</param>
/// <param name="PromptInitial">Initial value for prompt-kind dialogs.</param>
/// <param name="Placeholder">Placeholder shown when the prompt input is empty.</param>
/// <param name="Content">For <see cref="DialogKind.Custom"/>: the body fragment to
/// render inside <c>.modal-body</c>; <c>null</c> for confirm/prompt.</param>
/// <param name="Size">Optional Bootstrap modal-dialog size modifier class applied
/// to the <c>modal-dialog</c> element (e.g. <c>modal-lg</c>); <c>null</c> = default.</param>
/// <param name="ConfirmLabel">Explicit verb for the confirm button. <c>null</c> or
/// blank falls back to the kind/danger default ("Save" for a prompt, "Delete" for a
/// danger confirm, "Confirm" otherwise).</param>
public record DialogState(
string Title,
DialogKind Kind,
string Body,
bool Danger,
DialogSeverity Severity,
string PromptInitial,
string? Placeholder,
RenderFragment? Content = null,
string? Size = null);
string? Size = null,
string? ConfirmLabel = null);
public enum DialogKind
{
@@ -162,3 +183,39 @@ public enum DialogKind
Prompt,
Custom
}
/// <summary>
/// How consequential the confirmed action is — drives the confirm button's colour.
/// </summary>
/// <remarks>
/// Deferred-work register row 75. The dialog surface used to carry a two-state
/// <c>bool danger</c>, which forced every consequential-but-not-destructive action to
/// choose between looking like a delete and looking like the safe default. The Secured
/// Writes page is where that bit them: <em>Approve</em> and <em>Reject</em> both rendered
/// as identical red buttons, so at the moment of a two-person-control decision the colour
/// carried no discriminating information at all — and the affirmative half of the control
/// was painted with the delete affordance.
/// <para>
/// <see cref="Caution"/> is the middle rung that was missing. It is for actions that are
/// consequential and worth a beat of hesitation, but are not destroying anything: the
/// canonical case is a write to live plant equipment.
/// </para>
/// </remarks>
public enum DialogSeverity
{
/// <summary>Ordinary confirmation — renders <c>btn-primary</c>.</summary>
Normal,
/// <summary>
/// Consequential but not destructive — renders <c>btn-warning</c>. Does NOT inherit
/// the "Delete" label fallback; a caution dialog with no explicit label reads
/// "Confirm".
/// </summary>
Caution,
/// <summary>
/// Destructive or irreversible — renders <c>btn-danger</c>, and is the only severity
/// that falls back to the "Delete" verb (see <c>DialogHost.ConfirmLabel</c>).
/// </summary>
Destructive
}
src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Shared/IDialogService.cs
+42 -3
View File
@@ -19,10 +19,49 @@ public interface IDialogService
/// <param name="title">Modal title text.</param>
/// <param name="message">Body text shown to the user.</param>
/// <param name="danger">When <c>true</c>, the confirm button renders in
/// <c>btn-danger</c> styling with the label "Delete"; otherwise a primary
/// "Confirm" button is shown.</param>
/// <c>btn-danger</c> styling; otherwise a primary button is shown. This
/// controls <em>styling only</em> — it does not name the action. For an action that
/// is consequential but NOT destructive, use the <see cref="DialogSeverity"/>
/// overload and pass <see cref="DialogSeverity.Caution"/> rather than reaching for
/// red (deferred-work register row 75).</param>
/// <param name="confirmLabel">Verb rendered on the confirm button. When
/// <c>null</c> or blank the button falls back to "Delete" for a danger
/// dialog and "Confirm" otherwise. <strong>Any destructive action that is
/// not literally a delete must pass this</strong> — otherwise the red button
/// on, say, a failover or a discard reads "Delete", which is simply wrong
/// (deferred-work register row 28). See the label table in
/// <c>Component-CentralUI.md</c> → Dialog Service.</param>
/// <returns>A task that resolves to <c>true</c> when the user confirms, or <c>false</c> when cancelled.</returns>
Task<bool> ConfirmAsync(string title, string message, bool danger = false);
Task<bool> ConfirmAsync(string title, string message, bool danger = false, string? confirmLabel = null);
/// <summary>
/// Shows a confirmation dialog with an explicit <see cref="DialogSeverity"/>, for
/// actions the <c>danger</c> boolean cannot describe.
/// </summary>
/// <remarks>
/// Deferred-work register row 75. This is an ADDITIVE overload, deliberately: the
/// boolean form stays the signature for the ~34 existing call sites, all of which are
/// genuinely two-state. Changing the boolean's type instead would have rippled into
/// pages this change has no business touching, and adding a fifth optional parameter
/// to the boolean form would have silently un-matched every existing four-argument
/// NSubstitute stub — the exact gotcha register row 28 recorded (a stub that stops
/// matching returns a null Task and the caller NREs, with no compile error). A
/// separate overload cannot do either.
/// <para>
/// Reach for this only when <see cref="DialogSeverity.Caution"/> is what you mean.
/// For Normal / Destructive the boolean form says the same thing more briefly.
/// </para>
/// </remarks>
/// <param name="title">Modal title text.</param>
/// <param name="message">Body text shown to the user.</param>
/// <param name="severity">How consequential the action is; selects the confirm
/// button's styling.</param>
/// <param name="confirmLabel">Verb rendered on the confirm button. Only
/// <see cref="DialogSeverity.Destructive"/> falls back to "Delete" when this is
/// blank; the other severities fall back to "Confirm". The row-28 rule still
/// applies — name the action.</param>
/// <returns>A task that resolves to <c>true</c> when the user confirms, or <c>false</c> when cancelled.</returns>
Task<bool> ConfirmAsync(string title, string message, DialogSeverity severity, string? confirmLabel = null);
/// <summary>
/// Shows a single-line text prompt and resolves to the entered value, or
src/ZB.MOM.WW.ScadaBridge.Commons/Entities/Transport/BundleUnlockAttempt.cs
+42
View File
@@ -0,0 +1,42 @@
namespace ZB.MOM.WW.ScadaBridge.Commons.Entities.Transport;
/// <summary>
/// One recorded bundle-unlock passphrase attempt, written to the central
/// <c>BundleUnlockAttempt</c> table. Rows form the trailing-hour sliding window the
/// Transport component's per-client unlock throttle
/// (<c>TransportOptions.MaxUnlockAttemptsPerIpPerHour</c>) is evaluated against.
/// </summary>
/// <remarks>
/// <para>
/// Register row 52: the throttle's counters used to be per-node and per-process, so the
/// effective fleet cap was N x limit per client and a node restart cleared the window.
/// Central MS SQL is the store both central nodes already share, so putting the window
/// there makes the cap fleet-wide and restart-durable without new infrastructure.
/// </para>
/// <para>
/// Operational history, NOT audit — the table is non-partitioned, lives on the standard
/// <c>[PRIMARY]</c> filegroup and carries no DB-role restriction. Rows are pruned
/// opportunistically once they age past the window; they are not a durable record of
/// who tried to open which bundle (that is the <c>BundleImportUnlockFailed</c> audit row's
/// job, and note this table deliberately records no bundle identity at all).
/// </para>
/// <para>
/// Persistence-ignorant POCO; the EF Core mapping lives in the Configuration Database
/// component. <see cref="AttemptedAtUtc"/> is UTC, like every timestamp in the system.
/// </para>
/// </remarks>
public sealed class BundleUnlockAttempt
{
/// <summary>Surrogate identity key assigned by the store.</summary>
public long Id { get; set; }
/// <summary>
/// The throttle bucket the attempt was counted against — the caller's remote IP, or
/// the shared <c>&lt;unattributed&gt;</c> bucket when no address could be attributed.
/// Sized for an IPv6 literal with a scope id.
/// </summary>
public required string ClientKey { get; set; }
/// <summary>The UTC instant at which the unlock attempt was registered.</summary>
public DateTime AttemptedAtUtc { get; set; }
}
src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IBundleUnlockAttemptRepository.cs
+46
View File
@@ -0,0 +1,46 @@
namespace ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
/// <summary>
/// Register row 52: read/write access to the central <c>BundleUnlockAttempt</c> table,
/// the fleet-wide sliding window behind the Transport component's per-client bundle-unlock
/// throttle. Both central nodes share this store, so the documented
/// <c>MaxUnlockAttemptsPerIpPerHour</c> cap is a fleet cap rather than a per-process one.
/// </summary>
/// <remarks>
/// The throttle is a best-effort rate cap layered over an already-authenticated,
/// role-gated surface. Callers are expected to treat a store failure as fail-open (the
/// availability of legitimate imports outranks the cap), so no method here is required to
/// be transactional with respect to the caller's own work.
/// </remarks>
public interface IBundleUnlockAttemptRepository
{
/// <summary>
/// Counts the attempts recorded for <paramref name="clientKey"/> at or after
/// <paramref name="sinceUtc"/> — the trailing-window depth the cap is compared against.
/// </summary>
/// <param name="clientKey">Throttle bucket (client IP, or the shared unattributed bucket).</param>
/// <param name="sinceUtc">Inclusive UTC lower bound of the trailing window.</param>
/// <param name="ct">Cancellation token.</param>
/// <returns>The number of attempts inside the window.</returns>
Task<int> CountAttemptsSinceAsync(string clientKey, DateTime sinceUtc, CancellationToken ct = default);
/// <summary>
/// Appends one attempt row for <paramref name="clientKey"/>.
/// </summary>
/// <param name="clientKey">Throttle bucket (client IP, or the shared unattributed bucket).</param>
/// <param name="attemptedAtUtc">UTC instant of the attempt.</param>
/// <param name="ct">Cancellation token.</param>
/// <returns>A task that completes when the row is committed.</returns>
Task RecordAttemptAsync(string clientKey, DateTime attemptedAtUtc, CancellationToken ct = default);
/// <summary>
/// Deletes attempt rows older than <paramref name="beforeUtc"/>, up to
/// <paramref name="maxRows"/> in one bounded pass. Bounded so an opportunistic prune on
/// a request path can never turn into an unbounded delete.
/// </summary>
/// <param name="beforeUtc">Exclusive UTC upper bound — rows strictly older than this are eligible.</param>
/// <param name="maxRows">Maximum rows to delete in this pass; must be positive.</param>
/// <param name="ct">Cancellation token.</param>
/// <returns>The number of rows deleted.</returns>
Task<int> PruneAttemptsOlderThanAsync(DateTime beforeUtc, int maxRows, CancellationToken ct = default);
}
src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Transport/IBundleSessionStore.cs
+9
View File
@@ -18,6 +18,15 @@ public interface IBundleSessionStore
/// <summary>Removes all sessions whose expiry has passed.</summary>
void EvictExpired();
/// <summary>
/// Register row 53: the number of sessions currently held that have not yet expired.
/// Backs the <c>scadabridge.central.transport.import_sessions.open</c> gauge, so it
/// must be cheap and side-effect-free (it is read from the metrics collection thread)
/// and must NOT count expired-but-unswept entries — otherwise the gauge would read
/// as "at the cap" for up to a sweep interval after a burst drained.
/// </summary>
int OpenSessionCount { get; }
/// <summary>
/// T-003: returns the current unlock-failure count for a bundle keyed by its
/// content hash. The counter is server-owned so a second tab / CLI caller
src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Artifacts/ExternalSystemArtifact.cs
+33 -1
View File
@@ -1,9 +1,41 @@
namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
/// <summary>
/// A single external HTTP system, flattened for deployment to a site.
/// </summary>
/// <param name="Name">Unique system name — the key scripts call it by.</param>
/// <param name="EndpointUrl">Base URL of the system's HTTP endpoint.</param>
/// <param name="AuthType">Authentication type identifier (e.g. <c>ApiKey</c>, <c>BasicAuth</c>).</param>
/// <param name="AuthConfiguration">JSON-serialized authentication configuration for <paramref name="AuthType"/>.</param>
/// <param name="MethodDefinitionsJson">JSON-serialized method list, or <c>null</c> when the system defines none.</param>
/// <param name="TimeoutSeconds">
/// Per-system HTTP round-trip timeout in seconds. <c>0</c> = unset — the gateway's configured
/// <c>DefaultHttpTimeout</c> applies.
/// </param>
/// <param name="MaxRetries">
/// Store-and-Forward retry budget for this system's cached calls. <c>0</c> = unset — the engine's
/// <c>DefaultMaxRetries</c> applies.
/// <para>
/// <b>0 means "unset", not "no limit", at THIS layer</b> — the inverse of
/// <c>StoreAndForwardMessage.MaxRetries</c>, where a stored 0 means unbounded retry. The two
/// meanings are reconciled in <c>ExternalSystemClient.CachedCallAsync</c>, which passes a 0
/// through as <c>null</c> so the bounded engine default applies rather than an unbounded retry
/// loop. Keeping "unset" as 0 here (rather than a nullable) matches
/// <see cref="DatabaseConnectionArtifact"/>, the non-nullable
/// <c>ExternalSystemDefinition.MaxRetries</c> entity property this is projected from, and the
/// proto3 scalar it is carried on.
/// </para>
/// </param>
/// <param name="RetryDelay">
/// Fixed interval between Store-and-Forward retry sweeps for this system's cached calls.
/// <see cref="TimeSpan.Zero"/> = unset — the engine's <c>DefaultRetryInterval</c> applies.
/// </param>
public record ExternalSystemArtifact(
string Name,
string EndpointUrl,
string AuthType,
string? AuthConfiguration,
string? MethodDefinitionsJson,
int TimeoutSeconds = 0);
int TimeoutSeconds = 0,
int MaxRetries = 0,
TimeSpan RetryDelay = default);
src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Deployment/EnsureInstanceStartedCommand.cs
+29
View File
@@ -0,0 +1,29 @@
namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
/// <summary>
/// Site-internal request to make sure an already-persisted instance is actually RUNNING.
/// Routes via the deployment-manager singleton proxy so the Instance Actor is only ever
/// created under the singleton, whichever node raised the request.
/// <para>
/// Emitted by <c>SiteReconciliationActor</c> after its node-STARTUP pass fetches and stores
/// a config the node was missing. The reconcile heal path writes the FILE; without this the
/// runtime was left untouched, so an instance whose config was absent at boot (lost volume,
/// first boot after a restore, rehomed data dir) stayed dead until the NEXT restart, whose
/// SQLite load finally started it.
/// </para>
/// <para>
/// <b>Fire-and-forget and idempotent.</b> There is no reply: the handler no-ops when the
/// instance is already running, when an in-flight deploy will create it anyway, or when the
/// durable row says the instance is disabled. The stored row — not this message — is the
/// authority on both the config body and enabled/disabled state, so a disable that landed
/// after the reconcile fetch still wins.
/// </para>
/// </summary>
/// <param name="InstanceUniqueName">Unique name of the instance that should be running.</param>
/// <param name="Reason">
/// Short human-readable provenance for the request (e.g. <c>"startup reconcile"</c>), carried
/// verbatim into the log line and the operational site event so the heal is not silent.
/// </param>
public record EnsureInstanceStartedCommand(
string InstanceUniqueName,
string Reason);
src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SiteHealthReport.cs
+44 -5
View File
@@ -103,12 +103,23 @@ public record SiteHealthReport(
/// or <see langword="null"/> when replication is not wired on this node.
/// </summary>
/// <remarks>
/// <para>
/// Nullable on purpose, and <see langword="false"/> is NOT the same as null.
/// Replication ships default-OFF, so a site node with no peer configured reports
/// <see langword="false"/> — that is a healthy, expected state, not an outage. Only
/// a node that was configured with a peer and is reporting <see langword="false"/>
/// is degraded, and distinguishing those two cases is the operator's job, not this
/// field's.
/// <b>Null means "no reading"</b> — replication is not configured on this node
/// (the default-OFF posture), or the reporter has not run yet.
/// <b><see langword="false"/> means "configured, and the peer session is down"</b>,
/// which is a genuine fault worth investigating.
/// </para>
/// <para>
/// The site side must never publish <see langword="false"/> for an unconfigured node
/// (deferred-work register row 80): the replication engine is registered
/// unconditionally and answers <c>Connected=false</c> while inert, so passing that
/// through made every deliberately-unreplicated site look like a broken replicated
/// one, and starved central's null branches (conditional KPI emission,
/// sustained-badge anchor exclusion) of the null they were built for.
/// <c>LocalDbReplicationStatusReporter</c> gates on the Host-supplied
/// <c>SiteLocalDbSetup.ReplicationIsConfigured</c> to keep that distinction honest.
/// </para>
/// </remarks>
public bool? LocalDbReplicationConnected { get; init; }
@@ -134,6 +145,34 @@ public record SiteHealthReport(
/// 0 so existing producers/tests that don't populate it stay valid.
/// </summary>
public long SiteStreamAlarmDropCount { get; init; }
/// <summary>
/// Whether this node spawned its <c>SiteReconciliationActor</c> at boot — the per-node
/// startup self-heal that lets a standby which was DOWN during a deploy recover its
/// deployed inventory (and, since register row 67, restart the Instance Actors a
/// boot-time load missed).
/// </summary>
/// <remarks>
/// <para>
/// Tri-state, and the distinction is the whole point (deferred-work register row 73).
/// <b>Null = not reported</b>: a central self-report, where the notion does not apply,
/// or a site node running a pre-row-73 build. <b><see langword="true"/></b> = the
/// reconciler is running. <b><see langword="false"/></b> = the node booted WITHOUT it,
/// because no <c>IDeploymentConfigFetcher</c> was resolvable.
/// </para>
/// <para>
/// That false case used to be invisible: <c>AkkaHostedService</c> logged a single boot
/// Warning and spawned nothing, so a misconfigured site reported itself perfectly
/// healthy forever while being unable to self-heal. Central badges it rather than
/// failing the site — replication remains the primary path, so the node is degraded,
/// not down.
/// </para>
/// <para>
/// Stamped once at boot by the Host (which owns the spawn decision) via
/// <c>ISiteHealthCollector.SetStartupReconcilerActive</c>; it is not polled.
/// </para>
/// </remarks>
public bool? StartupReconcilerActive { get; init; }
}
/// <summary>
src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Management/AuditLogCommands.cs
+19
View File
@@ -1,6 +1,25 @@
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Audit;
namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.Management;
public record QueryAuditLogCommand(
string? User = null, string? EntityType = null, string? Action = null,
DateTimeOffset? From = null, DateTimeOffset? To = null,
int Page = 1, int PageSize = 50);
/// <summary>
/// Result wrapper for <see cref="QueryAuditLogCommand"/>.
/// </summary>
/// <remarks>
/// This record exists specifically so the handler does not return the repository's
/// <c>(IReadOnlyList&lt;AuditLogEntry&gt; Entries, int TotalCount)</c> ValueTuple directly.
/// A ValueTuple exposes <c>Item1</c>/<c>Item2</c> as public <em>fields</em>, and
/// <see cref="System.Text.Json"/> serializes only properties unless
/// <c>JsonSerializerOptions.IncludeFields</c> is set — so a tuple result serialized to
/// the literal string <c>{}</c> and every <c>audit-config query</c> looked empty no
/// matter how many rows the query matched (deferred-work register row 46). Mirrors the
/// <c>SecuredWriteListResult</c> shape used by the other paged management query.
/// </remarks>
/// <param name="Entries">The page of configuration-change audit entries.</param>
/// <param name="TotalCount">Total number of rows matching the filters (unpaged).</param>
public record AuditLogQueryResult(IReadOnlyList<AuditLogEntry> Entries, int TotalCount);
src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Management/ExternalSystemCommands.cs
+17 -2
View File
@@ -2,8 +2,23 @@ namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.Management;
public record ListExternalSystemsCommand;
public record GetExternalSystemCommand(int ExternalSystemId);
public record CreateExternalSystemCommand(string Name, string EndpointUrl, string AuthType, string? AuthConfiguration, int? TimeoutSeconds = null);
public record UpdateExternalSystemCommand(int ExternalSystemId, string Name, string EndpointUrl, string AuthType, string? AuthConfiguration, int? TimeoutSeconds = null);
/// <summary>
/// Creates an external HTTP system.
/// </summary>
/// <remarks>
/// <paramref name="MaxRetries"/> and <paramref name="RetryDelay"/> are the Store-and-Forward
/// retry policy applied to this system's <c>ExternalSystem.CachedCall</c> buffering. Both are
/// nullable on the wire so "omit" is distinguishable from "set to 0"; a null (or an explicit 0)
/// stores the entity's unset value, which the site gateway translates into the engine's
/// <c>DefaultMaxRetries</c>/<c>DefaultRetryInterval</c>.
/// </remarks>
public record CreateExternalSystemCommand(string Name, string EndpointUrl, string AuthType, string? AuthConfiguration, int? TimeoutSeconds = null, int? MaxRetries = null, TimeSpan? RetryDelay = null);
/// <summary>
/// Updates an external HTTP system. Nullable members follow the "omit to preserve" convention —
/// a null leaves the stored value untouched.
/// </summary>
public record UpdateExternalSystemCommand(int ExternalSystemId, string Name, string EndpointUrl, string AuthType, string? AuthConfiguration, int? TimeoutSeconds = null, int? MaxRetries = null, TimeSpan? RetryDelay = null);
public record DeleteExternalSystemCommand(int ExternalSystemId);
// External System Methods
src/ZB.MOM.WW.ScadaBridge.Commons/Observability/ScadaBridgeTelemetry.cs
+236 -11
View File
@@ -1,4 +1,5 @@
using System.Diagnostics.Metrics;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
namespace ZB.MOM.WW.ScadaBridge.Commons.Observability;
@@ -93,6 +94,38 @@ public static class ScadaBridgeTelemetry
Meter.CreateCounter<long>("scadabridge.site.stream.events_dropped", unit: "1",
description: "Events evicted from a site gRPC stream's bounded send channel, tagged by stream kind.");
/// <summary>
/// Register row 66: incremented each time <c>BundleSessionStore.Open</c> refuses a
/// bundle-import session because <c>TransportOptions.MaxConcurrentImportSessions</c>
/// (default 8) is genuinely exhausted — i.e. the store was still at the cap AFTER
/// expired sessions were reclaimed. Reclaim-then-succeed and re-opening an already
/// open session id are deliberately NOT counted: neither turns an operator away.
///
/// Row 53 shipped the companion open-sessions gauge and a Warning log, but a log line
/// is not alertable. The pair is what distinguishes the two failure modes: rejections
/// climbing while the gauge oscillates below the cap is genuine concurrency (raise the
/// cap), whereas rejections climbing while the gauge sits pinned at the cap is a
/// session leak like register row 43's (find the leak — raising the cap only defers it).
/// </summary>
private static readonly Counter<long> _transportImportSessionsRejected =
Meter.CreateCounter<long>("scadabridge.central.transport.import_sessions.rejected", unit: "1",
description: "Bundle-import sessions rejected because the concurrent-import-session cap was exhausted.");
/// <summary>
/// Register row 34: incremented once per site-event-log purge tick that stopped at
/// <c>SiteEventLogOptions.MaxPurgedRowsPerTick</c> with expired rows still to delete.
///
/// A single hit is routine (a backlog is being worked through on accelerated catch-up
/// ticks) — what the series is for is the SHAPE. A short burst of hits that stops is a
/// backlog converging; hits that never stop mean the event log is being written faster
/// than one paced tick can purge, i.e. the ceiling or the catch-up interval is now
/// mis-sized for this site's event rate. The paired log line is rate-limited and so
/// cannot answer that question on its own.
/// </summary>
private static readonly Counter<long> _siteEventLogPurgeCeilingHits =
Meter.CreateCounter<long>("scadabridge.site.event_log.purge_ceiling_hits", unit: "1",
description: "Site event log purge ticks that stopped at the per-tick delete ceiling with work remaining.");
// ---------------- Histograms ----------------
/// <summary>
@@ -109,8 +142,27 @@ public static class ScadaBridgeTelemetry
// ---------------- Observable gauges ----------------
/// <summary>Current count of open site connections, mutated via <see cref="Interlocked"/>.</summary>
private static long _siteConnectionsUp;
/// <summary>
/// Current count of open central→site gRPC subscriber streams, mutated via
/// <see cref="Interlocked"/>. See <see cref="_siteStreamSubscribersActive"/>'s gauge for
/// why this is NOT a data-connection count.
/// </summary>
private static long _siteStreamSubscriberCount;
/// <summary>
/// Register row 54: last-known state of every data connection on this site node, keyed by
/// connection name. Written by the DCL at its four state-transition points and read by the
/// data-connection gauges below.
/// </summary>
/// <remarks>
/// A static registry, matching <see cref="_siteStreamSubscriberCount"/>'s precedent, rather
/// than a DI-injected holder: the DCL actors already receive <c>ISiteHealthCollector</c>
/// through their constructors, and threading a second dependency through both actor
/// constructors would ripple into every test that builds them. A site node hosts exactly
/// one DCL, so a process-global registry and a per-node one are the same thing here.
/// </remarks>
private static readonly System.Collections.Concurrent.ConcurrentDictionary<string, ConnectionHealth>
_dataConnectionStates = new(StringComparer.Ordinal);
/// <summary>Current count of running per-site live-alarm aggregators, mutated via <see cref="Interlocked"/>.</summary>
private static long _liveAlarmAggregatorsActive;
@@ -118,12 +170,61 @@ public static class ScadaBridgeTelemetry
/// <summary>Provider that yields the live StoreAndForward queue depth; set by a later task.</summary>
private static Func<long>? _queueDepthProvider;
/// <summary>
/// Register row 53: provider that yields the number of currently open (unexpired)
/// Transport bundle-import sessions, registered by the Transport component's
/// session-eviction hosted service. Null on a site node (Transport is central-only),
/// where the gauge simply reports 0.
/// </summary>
private static Func<long>? _importSessionCountProvider;
#pragma warning disable IDE0052 // Held to keep the observable gauges alive for the meter's lifetime.
/// <summary>Gauge reporting the number of currently open site connections.</summary>
private static readonly ObservableGauge<long> _siteConnectionUp =
Meter.CreateObservableGauge<long>("scadabridge.site.connection.up",
() => Interlocked.Read(ref _siteConnectionsUp),
description: "Number of currently open site connections.");
/// <summary>
/// Gauge reporting the number of open central→site gRPC subscriber streams.
/// </summary>
/// <remarks>
/// Register row 54 RENAMED this from <c>scadabridge.site.connection.up</c>
/// (<c>scadabridge_site_connection_up</c> in Prometheus). The old name read as a
/// data-connection health signal, but it has only ever counted open subscriber streams
/// on <c>SiteStreamGrpcServer</c> — Debug View and the site-wide alarm feed. It therefore
/// sits at 0 on a perfectly healthy site that nobody is watching, and an alert written
/// against the old name fired on an idle operator console rather than a broken device.
/// For real data-connection health use the <c>scadabridge.site.data_connections.*</c>
/// gauges below. Counts both stream kinds together (no <c>stream</c> tag).
/// </remarks>
private static readonly ObservableGauge<long> _siteStreamSubscribersActive =
Meter.CreateObservableGauge<long>("scadabridge.site.stream.subscribers.active",
() => Interlocked.Read(ref _siteStreamSubscriberCount),
unit: "1",
description: "Open central→site gRPC subscriber streams (Debug View + site-wide alarm feed). NOT data connections.");
/// <summary>
/// Register row 54: gauge reporting how many of this site node's data connections are
/// currently in <see cref="ConnectionHealth.Connected"/> — the real data-connection
/// signal the misnamed stream-subscriber gauge was being mistaken for.
/// </summary>
/// <remarks>
/// Read together with <see cref="_dataConnectionsTotal"/>: <c>connected &lt; total</c> is the
/// alertable condition, and neither number alone distinguishes "one device down" from
/// "nothing configured". Expect **0 on the standby node** — connections are opened by the
/// deployment path on the active node only, so a standby reporting 0 is healthy, not a
/// fault. Alert on the active node, or on <c>max()</c> across the pair.
/// </remarks>
private static readonly ObservableGauge<long> _dataConnectionsConnected =
Meter.CreateObservableGauge<long>("scadabridge.site.data_connections.connected",
CountDataConnections,
unit: "1",
description: "Data connections on this site node currently in the Connected state.");
/// <summary>
/// Register row 54: gauge reporting how many data connections this site node knows about,
/// in any state. The denominator for <see cref="_dataConnectionsConnected"/>.
/// </summary>
private static readonly ObservableGauge<long> _dataConnectionsTotal =
Meter.CreateObservableGauge<long>("scadabridge.site.data_connections.total",
() => (long)_dataConnectionStates.Count,
unit: "1",
description: "Data connections known to this site node, in any state.");
/// <summary>Gauge reporting the current StoreAndForward queue depth via the registered provider.</summary>
private static readonly ObservableGauge<long> _storeAndForwardQueueDepth =
@@ -138,6 +239,20 @@ public static class ScadaBridgeTelemetry
() => Interlocked.Read(ref _liveAlarmAggregatorsActive),
unit: "1",
description: "Number of per-site live-alarm aggregators currently running on the active central node.");
/// <summary>
/// Register row 53: gauge reporting the number of open Transport bundle-import
/// sessions on this central node. Each open session pins a fully decrypted bundle
/// in memory and consumes one of <c>TransportOptions.MaxConcurrentImportSessions</c>
/// (default 8), so a series that plateaus at the cap is the signature of leaked
/// sessions — the failure mode register row 43 could only be inferred from rotating
/// test failures because nothing counted them.
/// </summary>
private static readonly ObservableGauge<long> _transportImportSessionsOpen =
Meter.CreateObservableGauge<long>("scadabridge.central.transport.import_sessions.open",
() => Volatile.Read(ref _importSessionCountProvider) is { } p ? p() : 0L,
unit: "1",
description: "Number of currently open bundle-import sessions on this central node.");
#pragma warning restore IDE0052
// ---------------- Emit helpers ----------------
@@ -164,11 +279,83 @@ public static class ScadaBridgeTelemetry
/// <summary>Records one resync whose ack window expired.</summary>
public static void RecordSfResyncAckMissing() => _sfResyncAckMissing.Add(1);
/// <summary>Records that a site connection opened (increments the up-count gauge).</summary>
public static void SiteConnectionOpened() => Interlocked.Increment(ref _siteConnectionsUp);
/// <summary>
/// Records one bundle-import session rejected on a genuine concurrent-session cap hit
/// (register row 66). Call ONLY from the post-eviction recheck in
/// <c>BundleSessionStore.Open</c> — counting the pre-eviction check would inflate the
/// series with reclaim-then-succeed cases that never turned an operator away.
/// </summary>
public static void RecordImportSessionRejected() => _transportImportSessionsRejected.Add(1);
/// <summary>Records that a site connection closed (decrements the up-count gauge).</summary>
public static void SiteConnectionClosed() => Interlocked.Decrement(ref _siteConnectionsUp);
/// <summary>
/// Records one site-event-log purge tick that stopped at its per-tick delete ceiling
/// (register row 34). Call ONLY when rows genuinely remain to be purged — a tick that
/// finished the work and happened to land exactly on the ceiling has nothing to report.
/// </summary>
public static void RecordSiteEventLogPurgeCeilingHit() => _siteEventLogPurgeCeilingHits.Add(1);
/// <summary>
/// Records that a central→site gRPC subscriber stream opened (increments
/// <c>scadabridge.site.stream.subscribers.active</c>). Renamed from
/// <c>SiteConnectionOpened</c> by register row 54 — it never counted data connections.
/// </summary>
public static void SiteStreamSubscriberOpened() => Interlocked.Increment(ref _siteStreamSubscriberCount);
/// <summary>
/// Records that a central→site gRPC subscriber stream closed (decrements
/// <c>scadabridge.site.stream.subscribers.active</c>). Renamed from
/// <c>SiteConnectionClosed</c> by register row 54.
/// </summary>
public static void SiteStreamSubscriberClosed() => Interlocked.Decrement(ref _siteStreamSubscriberCount);
/// <summary>
/// Records the current state of one data connection, feeding the
/// <c>scadabridge.site.data_connections.*</c> gauges (register row 54). Called by the DCL
/// at each connection state transition; safe to call repeatedly with the same value.
/// </summary>
/// <param name="connectionName">The data connection's name (unique per site node).</param>
/// <param name="state">The connection's new state.</param>
public static void RecordDataConnectionState(string connectionName, ConnectionHealth state)
{
if (string.IsNullOrEmpty(connectionName))
{
return;
}
_dataConnectionStates[connectionName] = state;
}
/// <summary>
/// Drops a data connection from the gauges (register row 54). Called when the DCL removes
/// a connection, so an undeployed connection stops counting toward the total rather than
/// lingering as a permanently-not-connected row that would alert forever.
/// </summary>
/// <param name="connectionName">The data connection's name.</param>
public static void RemoveDataConnection(string connectionName)
{
if (string.IsNullOrEmpty(connectionName))
{
return;
}
_dataConnectionStates.TryRemove(connectionName, out _);
}
/// <summary>Counts data connections currently in <see cref="ConnectionHealth.Connected"/>.</summary>
/// <returns>The connected count.</returns>
private static long CountDataConnections()
{
long connected = 0;
foreach (var state in _dataConnectionStates.Values)
{
if (state == ConnectionHealth.Connected)
{
connected++;
}
}
return connected;
}
/// <summary>Records that a per-site live-alarm aggregator started (increments the active-aggregator gauge).</summary>
public static void LiveAlarmAggregatorStarted() => Interlocked.Increment(ref _liveAlarmAggregatorsActive);
@@ -242,4 +429,42 @@ public static class ScadaBridgeTelemetry
// delegate, so a stale stop cannot clobber a successor's provider.
Interlocked.CompareExchange(ref _queueDepthProvider, null, provider);
}
/// <summary>
/// Register row 53: registers the provider the open-import-sessions gauge reads on
/// each observation. Called by the Transport component's session-eviction hosted
/// service on start. A null provider is ignored so the gauge falls back to 0.
/// The callback is invoked synchronously on the metrics collection thread, so it
/// must stay cheap — the Transport implementation reads a
/// <see cref="System.Collections.Concurrent.ConcurrentDictionary{TKey,TValue}"/> it
/// already owns.
/// </summary>
/// <param name="provider">A callback returning the current open-session count.</param>
public static void SetImportSessionCountProvider(Func<long> provider)
{
if (provider is null)
{
return;
}
Volatile.Write(ref _importSessionCountProvider, provider);
}
/// <summary>
/// Clears the open-import-sessions provider, but only if the currently registered
/// provider is the exact <paramref name="provider"/> delegate passed in
/// (reference-equal compare-and-clear), so a late stop of a superseded instance
/// cannot stomp a newer registrant. Mirrors <see cref="ClearQueueDepthProvider"/>.
/// </summary>
/// <param name="provider">The provider delegate to remove; ignored unless it is the
/// one currently registered.</param>
public static void ClearImportSessionCountProvider(Func<long> provider)
{
if (provider is null)
{
return;
}
Interlocked.CompareExchange(ref _importSessionCountProvider, null, provider);
}
}
src/ZB.MOM.WW.ScadaBridge.Commons/Types/Kpi/KpiMetrics.cs
+17
View File
@@ -120,5 +120,22 @@ public static class KpiMetrics
/// <summary>Summed store-and-forward buffer depth across all buffers.</summary>
public const string SfBufferDepth = "sfBufferDepth";
/// <summary>
/// Unacked LocalDb replication oplog depth (register row 33). Emitted ONLY by sites
/// that actually report a backlog — an unreplicated site reports null, and null means
/// "no reading", never zero, so no sample is written at all rather than a fabricated 0.
/// A gap in this series therefore means "not replicated / not reporting", which is
/// exactly what it should mean.
/// </summary>
public const string LocalDbOplogBacklog = "localDbOplogBacklog";
/// <summary>
/// LocalDb replication link state as 1/0 (register row 33). Emitted ONLY when the site
/// reports a value; null (replication not configured) writes no sample. Note that
/// <c>false</c> and null are different states here — false is a configured link that is
/// down, null is no link configured — which is why this cannot be coerced to 0.
/// </summary>
public const string LocalDbReplicationConnected = "localDbReplicationConnected";
}
}
src/ZB.MOM.WW.ScadaBridge.Commons/Types/Transport/BundleLockedException.cs
+44
View File
@@ -0,0 +1,44 @@
namespace ZB.MOM.WW.ScadaBridge.Commons.Types.Transport;
/// <summary>
/// T-003: thrown by <c>IBundleImporter.LoadAsync</c> when an encrypted bundle has
/// exceeded the configured failed-unlock attempt limit
/// (<c>TransportOptions.MaxUnlockAttemptsPerSession</c>, default 3). The lockout is tracked
/// server-side keyed by <see cref="BundleManifest.ContentHash"/>, so a second tab / CLI caller
/// re-uploading the same bytes hits the same counter and cannot side-step the limit.
/// <para>
/// Distinct from <see cref="BundleUnlockThrottledException"/> on purpose: this one means
/// "this BUNDLE has burned its strikes" and is keyed by the bundle's content hash; that one
/// means "this CLIENT has attempted too many unlocks, across any bundles" and is keyed by
/// the caller's IP.
/// </para>
/// <para>
/// Register row 51: lives in Commons rather than beside the importer for the same reason
/// <see cref="BundleUnlockThrottledException"/> does — it is part of the
/// <c>IBundleImporter.LoadAsync</c> contract, which is declared here, and the management
/// surface (which references Commons but not Transport) must be able to catch it to return
/// a clear, curated error instead of a generic internal-failure message plus a correlation id.
/// </para>
/// </summary>
public sealed class BundleLockedException : Exception
{
/// <summary>Number of recorded unlock failures for this bundle.</summary>
public int FailedAttempts { get; }
/// <summary>SHA-256 (hex) of the bundle's content bytes, the lockout's tracking key.</summary>
public string BundleContentHash { get; }
/// <summary>
/// Initializes a new <see cref="BundleLockedException"/>.
/// </summary>
/// <param name="bundleContentHash">SHA-256 hex from <see cref="BundleManifest.ContentHash"/>.</param>
/// <param name="failedAttempts">Number of failures recorded against this bundle.</param>
public BundleLockedException(string bundleContentHash, int failedAttempts)
: base(
$"Bundle is locked after {failedAttempts} failed unlock attempts. "
+ "Wait for the lockout window to expire or re-export the bundle to obtain a new content hash.")
{
BundleContentHash = bundleContentHash ?? throw new ArgumentNullException(nameof(bundleContentHash));
FailedAttempts = failedAttempts;
}
}
src/ZB.MOM.WW.ScadaBridge.Communication/CentralControlGrpc/CentralControl.cs
+91 -38
View File
@@ -72,7 +72,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
"ZXkYASABKAkSQQoFdmFsdWUYAiABKAsyMi5zY2FkYWJyaWRnZS5jZW50cmFs",
"Y29udHJvbC52MS5UYWdRdWFsaXR5Q291bnRzRHRvOgI4ASJQChFOb2RlU3Rh",
"dHVzTGlzdER0bxI7CgVub2RlcxgBIAMoCzIsLnNjYWRhYnJpZGdlLmNlbnRy",
"YWxjb250cm9sLnYxLk5vZGVTdGF0dXNEdG8iig4KE1NpdGVIZWFsdGhSZXBv",
"YWxjb250cm9sLnYxLk5vZGVTdGF0dXNEdG8iyQ4KE1NpdGVIZWFsdGhSZXBv",
"cnREdG8SDwoHc2l0ZV9pZBgBIAEoCRIXCg9zZXF1ZW5jZV9udW1iZXIYAiAB",
"KAMSNAoQcmVwb3J0X3RpbWVzdGFtcBgDIAEoCzIaLmdvb2dsZS5wcm90b2J1",
"Zi5UaW1lc3RhbXAScAoYZGF0YV9jb25uZWN0aW9uX3N0YXR1c2VzGAQgAygL",
@@ -106,42 +106,43 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
"YmxlVmFsdWUSQgoebG9jYWxfZGJfcmVwbGljYXRpb25fY29ubmVjdGVkGBsg",
"ASgLMhouZ29vZ2xlLnByb3RvYnVmLkJvb2xWYWx1ZRI7ChZsb2NhbF9kYl9v",
"cGxvZ19iYWNrbG9nGBwgASgLMhsuZ29vZ2xlLnByb3RvYnVmLkludDY0VmFs",
"dWUacgobRGF0YUNvbm5lY3Rpb25TdGF0dXNlc0VudHJ5EgsKA2tleRgBIAEo",
"CRJCCgV2YWx1ZRgCIAEoDjIzLnNjYWRhYnJpZGdlLmNlbnRyYWxjb250cm9s",
"LnYxLkNvbm5lY3Rpb25IZWFsdGhFbnVtOgI4ARpxChhUYWdSZXNvbHV0aW9u",
"Q291bnRzRW50cnkSCwoDa2V5GAEgASgJEkQKBXZhbHVlGAIgASgLMjUuc2Nh",
"ZGFicmlkZ2UuY2VudHJhbGNvbnRyb2wudjEuVGFnUmVzb2x1dGlvblN0YXR1",
"c0R0bzoCOAEaQgogU3RvcmVBbmRGb3J3YXJkQnVmZmVyRGVwdGhzRW50cnkS",
"CwoDa2V5GAEgASgJEg0KBXZhbHVlGAIgASgFOgI4ASJjChZTaXRlSGVhbHRo",
"UmVwb3J0QWNrRHRvEg8KB3NpdGVfaWQYASABKAkSFwoPc2VxdWVuY2VfbnVt",
"YmVyGAIgASgDEhAKCGFjY2VwdGVkGAMgASgIEg0KBWVycm9yGAQgASgJIosB",
"CgxIZWFydGJlYXREdG8SDwoHc2l0ZV9pZBgBIAEoCRIVCg1ub2RlX2hvc3Ru",
"YW1lGAIgASgJEhEKCWlzX2FjdGl2ZRgDIAEoCBItCgl0aW1lc3RhbXAYBCAB",
"KAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wEhEKCXN5bnRoZXRpYxgF",
"IAEoCCq9AQoUQ29ubmVjdGlvbkhlYWx0aEVudW0SIQodQ09OTkVDVElPTl9I",
"RUFMVEhfVU5TUEVDSUZJRUQQABIfChtDT05ORUNUSU9OX0hFQUxUSF9DT05O",
"RUNURUQQARIiCh5DT05ORUNUSU9OX0hFQUxUSF9ESVNDT05ORUNURUQQAhIg",
"ChxDT05ORUNUSU9OX0hFQUxUSF9DT05ORUNUSU5HEAMSGwoXQ09OTkVDVElP",
"Tl9IRUFMVEhfRVJST1IQBDKhBgoVQ2VudHJhbENvbnRyb2xTZXJ2aWNlEoMB",
"ChJTdWJtaXROb3RpZmljYXRpb24SNC5zY2FkYWJyaWRnZS5jZW50cmFsY29u",
"dHJvbC52MS5Ob3RpZmljYXRpb25TdWJtaXREdG8aNy5zY2FkYWJyaWRnZS5j",
"ZW50cmFsY29udHJvbC52MS5Ob3RpZmljYXRpb25TdWJtaXRBY2tEdG8SkgEK",
"F1F1ZXJ5Tm90aWZpY2F0aW9uU3RhdHVzEjkuc2NhZGFicmlkZ2UuY2VudHJh",
"bGNvbnRyb2wudjEuTm90aWZpY2F0aW9uU3RhdHVzUXVlcnlEdG8aPC5zY2Fk",
"YWJyaWRnZS5jZW50cmFsY29udHJvbC52MS5Ob3RpZmljYXRpb25TdGF0dXNS",
"ZXNwb25zZUR0bxJHChFJbmdlc3RBdWRpdEV2ZW50cxIbLnNpdGVzdHJlYW0u",
"QXVkaXRFdmVudEJhdGNoGhUuc2l0ZXN0cmVhbS5Jbmdlc3RBY2sSUAoVSW5n",
"ZXN0Q2FjaGVkVGVsZW1ldHJ5EiAuc2l0ZXN0cmVhbS5DYWNoZWRUZWxlbWV0",
"cnlCYXRjaBoVLnNpdGVzdHJlYW0uSW5nZXN0QWNrEoABCg1SZWNvbmNpbGVT",
"aXRlEjYuc2NhZGFicmlkZ2UuY2VudHJhbGNvbnRyb2wudjEuUmVjb25jaWxl",
"U2l0ZVJlcXVlc3REdG8aNy5zY2FkYWJyaWRnZS5jZW50cmFsY29udHJvbC52",
"MS5SZWNvbmNpbGVTaXRlUmVzcG9uc2VEdG8SfQoQUmVwb3J0U2l0ZUhlYWx0",
"aBIyLnNjYWRhYnJpZGdlLmNlbnRyYWxjb250cm9sLnYxLlNpdGVIZWFsdGhS",
"ZXBvcnREdG8aNS5zY2FkYWJyaWRnZS5jZW50cmFsY29udHJvbC52MS5TaXRl",
"SGVhbHRoUmVwb3J0QWNrRHRvElAKCUhlYXJ0YmVhdBIrLnNjYWRhYnJpZGdl",
"LmNlbnRyYWxjb250cm9sLnYxLkhlYXJ0YmVhdER0bxoWLmdvb2dsZS5wcm90",
"b2J1Zi5FbXB0eUIrqgIoWkIuTU9NLldXLlNjYWRhQnJpZGdlLkNvbW11bmlj",
"YXRpb24uR3JwY2IGcHJvdG8z"));
"dWUSPQoZc3RhcnR1cF9yZWNvbmNpbGVyX2FjdGl2ZRgdIAEoCzIaLmdvb2ds",
"ZS5wcm90b2J1Zi5Cb29sVmFsdWUacgobRGF0YUNvbm5lY3Rpb25TdGF0dXNl",
"c0VudHJ5EgsKA2tleRgBIAEoCRJCCgV2YWx1ZRgCIAEoDjIzLnNjYWRhYnJp",
"ZGdlLmNlbnRyYWxjb250cm9sLnYxLkNvbm5lY3Rpb25IZWFsdGhFbnVtOgI4",
"ARpxChhUYWdSZXNvbHV0aW9uQ291bnRzRW50cnkSCwoDa2V5GAEgASgJEkQK",
"BXZhbHVlGAIgASgLMjUuc2NhZGFicmlkZ2UuY2VudHJhbGNvbnRyb2wudjEu",
"VGFnUmVzb2x1dGlvblN0YXR1c0R0bzoCOAEaQgogU3RvcmVBbmRGb3J3YXJk",
"QnVmZmVyRGVwdGhzRW50cnkSCwoDa2V5GAEgASgJEg0KBXZhbHVlGAIgASgF",
"OgI4ASJjChZTaXRlSGVhbHRoUmVwb3J0QWNrRHRvEg8KB3NpdGVfaWQYASAB",
"KAkSFwoPc2VxdWVuY2VfbnVtYmVyGAIgASgDEhAKCGFjY2VwdGVkGAMgASgI",
"Eg0KBWVycm9yGAQgASgJIosBCgxIZWFydGJlYXREdG8SDwoHc2l0ZV9pZBgB",
"IAEoCRIVCg1ub2RlX2hvc3RuYW1lGAIgASgJEhEKCWlzX2FjdGl2ZRgDIAEo",
"CBItCgl0aW1lc3RhbXAYBCABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0",
"YW1wEhEKCXN5bnRoZXRpYxgFIAEoCCq9AQoUQ29ubmVjdGlvbkhlYWx0aEVu",
"dW0SIQodQ09OTkVDVElPTl9IRUFMVEhfVU5TUEVDSUZJRUQQABIfChtDT05O",
"RUNUSU9OX0hFQUxUSF9DT05ORUNURUQQARIiCh5DT05ORUNUSU9OX0hFQUxU",
"SF9ESVNDT05ORUNURUQQAhIgChxDT05ORUNUSU9OX0hFQUxUSF9DT05ORUNU",
"SU5HEAMSGwoXQ09OTkVDVElPTl9IRUFMVEhfRVJST1IQBDKhBgoVQ2VudHJh",
"bENvbnRyb2xTZXJ2aWNlEoMBChJTdWJtaXROb3RpZmljYXRpb24SNC5zY2Fk",
"YWJyaWRnZS5jZW50cmFsY29udHJvbC52MS5Ob3RpZmljYXRpb25TdWJtaXRE",
"dG8aNy5zY2FkYWJyaWRnZS5jZW50cmFsY29udHJvbC52MS5Ob3RpZmljYXRp",
"b25TdWJtaXRBY2tEdG8SkgEKF1F1ZXJ5Tm90aWZpY2F0aW9uU3RhdHVzEjku",
"c2NhZGFicmlkZ2UuY2VudHJhbGNvbnRyb2wudjEuTm90aWZpY2F0aW9uU3Rh",
"dHVzUXVlcnlEdG8aPC5zY2FkYWJyaWRnZS5jZW50cmFsY29udHJvbC52MS5O",
"b3RpZmljYXRpb25TdGF0dXNSZXNwb25zZUR0bxJHChFJbmdlc3RBdWRpdEV2",
"ZW50cxIbLnNpdGVzdHJlYW0uQXVkaXRFdmVudEJhdGNoGhUuc2l0ZXN0cmVh",
"bS5Jbmdlc3RBY2sSUAoVSW5nZXN0Q2FjaGVkVGVsZW1ldHJ5EiAuc2l0ZXN0",
"cmVhbS5DYWNoZWRUZWxlbWV0cnlCYXRjaBoVLnNpdGVzdHJlYW0uSW5nZXN0",
"QWNrEoABCg1SZWNvbmNpbGVTaXRlEjYuc2NhZGFicmlkZ2UuY2VudHJhbGNv",
"bnRyb2wudjEuUmVjb25jaWxlU2l0ZVJlcXVlc3REdG8aNy5zY2FkYWJyaWRn",
"ZS5jZW50cmFsY29udHJvbC52MS5SZWNvbmNpbGVTaXRlUmVzcG9uc2VEdG8S",
"fQoQUmVwb3J0U2l0ZUhlYWx0aBIyLnNjYWRhYnJpZGdlLmNlbnRyYWxjb250",
"cm9sLnYxLlNpdGVIZWFsdGhSZXBvcnREdG8aNS5zY2FkYWJyaWRnZS5jZW50",
"cmFsY29udHJvbC52MS5TaXRlSGVhbHRoUmVwb3J0QWNrRHRvElAKCUhlYXJ0",
"YmVhdBIrLnNjYWRhYnJpZGdlLmNlbnRyYWxjb250cm9sLnYxLkhlYXJ0YmVh",
"dER0bxoWLmdvb2dsZS5wcm90b2J1Zi5FbXB0eUIrqgIoWkIuTU9NLldXLlNj",
"YWRhQnJpZGdlLkNvbW11bmljYXRpb24uR3JwY2IGcHJvdG8z"));
descriptor = pbr::FileDescriptor.FromGeneratedCode(descriptorData,
new pbr::FileDescriptor[] { global::Google.Protobuf.WellKnownTypes.EmptyReflection.Descriptor, global::Google.Protobuf.WellKnownTypes.TimestampReflection.Descriptor, global::Google.Protobuf.WellKnownTypes.WrappersReflection.Descriptor, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor, },
new pbr::GeneratedClrTypeInfo(new[] {typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ConnectionHealthEnum), }, null, new pbr::GeneratedClrTypeInfo[] {
@@ -159,7 +160,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ConnectionEndpointMapDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ConnectionEndpointMapDto.Parser, new[]{ "Entries" }, null, null, null, new pbr::GeneratedClrTypeInfo[] { null, }),
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.TagQualityMapDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.TagQualityMapDto.Parser, new[]{ "Entries" }, null, null, null, new pbr::GeneratedClrTypeInfo[] { null, }),
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NodeStatusListDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NodeStatusListDto.Parser, new[]{ "Nodes" }, null, null, null, null),
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto.Parser, new[]{ "SiteId", "SequenceNumber", "ReportTimestamp", "DataConnectionStatuses", "TagResolutionCounts", "ScriptErrorCount", "AlarmEvaluationErrorCount", "StoreAndForwardBufferDepths", "DeadLetterCount", "DeployedInstanceCount", "EnabledInstanceCount", "DisabledInstanceCount", "NodeRole", "NodeHostname", "DataConnectionEndpoints", "DataConnectionTagQuality", "ParkedMessageCount", "ClusterNodes", "SiteAuditWriteFailures", "AuditRedactionFailure", "SiteAuditBacklog", "SiteEventLogWriteFailures", "OldestParkedMessageAgeSeconds", "ScriptQueueDepth", "ScriptBusyThreads", "ScriptOldestBusyAgeSeconds", "LocalDbReplicationConnected", "LocalDbOplogBacklog" }, null, null, null, new pbr::GeneratedClrTypeInfo[] { null, null, null, }),
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto.Parser, new[]{ "SiteId", "SequenceNumber", "ReportTimestamp", "DataConnectionStatuses", "TagResolutionCounts", "ScriptErrorCount", "AlarmEvaluationErrorCount", "StoreAndForwardBufferDepths", "DeadLetterCount", "DeployedInstanceCount", "EnabledInstanceCount", "DisabledInstanceCount", "NodeRole", "NodeHostname", "DataConnectionEndpoints", "DataConnectionTagQuality", "ParkedMessageCount", "ClusterNodes", "SiteAuditWriteFailures", "AuditRedactionFailure", "SiteAuditBacklog", "SiteEventLogWriteFailures", "OldestParkedMessageAgeSeconds", "ScriptQueueDepth", "ScriptBusyThreads", "ScriptOldestBusyAgeSeconds", "LocalDbReplicationConnected", "LocalDbOplogBacklog", "StartupReconcilerActive" }, null, null, null, new pbr::GeneratedClrTypeInfo[] { null, null, null, }),
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto.Parser, new[]{ "SiteId", "SequenceNumber", "Accepted", "Error" }, null, null, null, null),
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto.Parser, new[]{ "SiteId", "NodeHostname", "IsActive", "Timestamp", "Synthetic" }, null, null, null, null)
}));
@@ -4266,6 +4267,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
ScriptOldestBusyAgeSeconds = other.ScriptOldestBusyAgeSeconds;
LocalDbReplicationConnected = other.LocalDbReplicationConnected;
LocalDbOplogBacklog = other.LocalDbOplogBacklog;
StartupReconcilerActive = other.StartupReconcilerActive;
_unknownFields = pb::UnknownFieldSet.Clone(other._unknownFields);
}
@@ -4642,6 +4644,27 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
}
/// <summary>Field number for the "startup_reconciler_active" field.</summary>
public const int StartupReconcilerActiveFieldNumber = 29;
private static readonly pb::FieldCodec<bool?> _single_startupReconcilerActive_codec = pb::FieldCodec.ForStructWrapper<bool>(234);
private bool? startupReconcilerActive_;
/// <summary>
/// Whether this node spawned its SiteReconciliationActor at boot. Absent means
/// "not reported" — a central self-report (where it is meaningless) or a site
/// node on a pre-row-73 build. false means the node booted WITHOUT the startup
/// self-heal path, which is a misconfiguration a healthy-looking site would
/// otherwise hide entirely.
/// </summary>
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
public bool? StartupReconcilerActive {
get { return startupReconcilerActive_; }
set {
startupReconcilerActive_ = value;
}
}
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
public override bool Equals(object other) {
@@ -4685,6 +4708,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
if (!pbc::ProtobufEqualityComparers.BitwiseNullableDoubleEqualityComparer.Equals(ScriptOldestBusyAgeSeconds, other.ScriptOldestBusyAgeSeconds)) return false;
if (LocalDbReplicationConnected != other.LocalDbReplicationConnected) return false;
if (LocalDbOplogBacklog != other.LocalDbOplogBacklog) return false;
if (StartupReconcilerActive != other.StartupReconcilerActive) return false;
return Equals(_unknownFields, other._unknownFields);
}
@@ -4720,6 +4744,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
if (scriptOldestBusyAgeSeconds_ != null) hash ^= pbc::ProtobufEqualityComparers.BitwiseNullableDoubleEqualityComparer.GetHashCode(ScriptOldestBusyAgeSeconds);
if (localDbReplicationConnected_ != null) hash ^= LocalDbReplicationConnected.GetHashCode();
if (localDbOplogBacklog_ != null) hash ^= LocalDbOplogBacklog.GetHashCode();
if (startupReconcilerActive_ != null) hash ^= StartupReconcilerActive.GetHashCode();
if (_unknownFields != null) {
hash ^= _unknownFields.GetHashCode();
}
@@ -4837,6 +4862,9 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
if (localDbOplogBacklog_ != null) {
_single_localDbOplogBacklog_codec.WriteTagAndValue(output, LocalDbOplogBacklog);
}
if (startupReconcilerActive_ != null) {
_single_startupReconcilerActive_codec.WriteTagAndValue(output, StartupReconcilerActive);
}
if (_unknownFields != null) {
_unknownFields.WriteTo(output);
}
@@ -4946,6 +4974,9 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
if (localDbOplogBacklog_ != null) {
_single_localDbOplogBacklog_codec.WriteTagAndValue(ref output, LocalDbOplogBacklog);
}
if (startupReconcilerActive_ != null) {
_single_startupReconcilerActive_codec.WriteTagAndValue(ref output, StartupReconcilerActive);
}
if (_unknownFields != null) {
_unknownFields.WriteTo(ref output);
}
@@ -5034,6 +5065,9 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
if (localDbOplogBacklog_ != null) {
size += _single_localDbOplogBacklog_codec.CalculateSizeWithTag(LocalDbOplogBacklog);
}
if (startupReconcilerActive_ != null) {
size += _single_startupReconcilerActive_codec.CalculateSizeWithTag(StartupReconcilerActive);
}
if (_unknownFields != null) {
size += _unknownFields.CalculateSize();
}
@@ -5147,6 +5181,11 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
LocalDbOplogBacklog = other.LocalDbOplogBacklog;
}
}
if (other.startupReconcilerActive_ != null) {
if (startupReconcilerActive_ == null || other.StartupReconcilerActive != false) {
StartupReconcilerActive = other.StartupReconcilerActive;
}
}
_unknownFields = pb::UnknownFieldSet.MergeFrom(_unknownFields, other._unknownFields);
}
@@ -5305,6 +5344,13 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
}
break;
}
case 234: {
bool? value = _single_startupReconcilerActive_codec.Read(input);
if (startupReconcilerActive_ == null || value != false) {
StartupReconcilerActive = value;
}
break;
}
}
}
#endif
@@ -5463,6 +5509,13 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
}
break;
}
case 234: {
bool? value = _single_startupReconcilerActive_codec.Read(ref input);
if (startupReconcilerActive_ == null || value != false) {
StartupReconcilerActive = value;
}
break;
}
}
}
}
src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/CentralControlDtoMapper.cs
+2
View File
@@ -475,6 +475,7 @@ public static class CentralControlDtoMapper
ScriptOldestBusyAgeSeconds = msg.ScriptOldestBusyAgeSeconds,
LocalDbReplicationConnected = msg.LocalDbReplicationConnected,
LocalDbOplogBacklog = msg.LocalDbOplogBacklog,
StartupReconcilerActive = msg.StartupReconcilerActive,
};
foreach (var (name, health) in msg.DataConnectionStatuses)
@@ -649,6 +650,7 @@ public static class CentralControlDtoMapper
ScriptOldestBusyAgeSeconds = dto.ScriptOldestBusyAgeSeconds,
LocalDbReplicationConnected = dto.LocalDbReplicationConnected,
LocalDbOplogBacklog = dto.LocalDbOplogBacklog,
StartupReconcilerActive = dto.StartupReconcilerActive,
};
}
src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteCommandDtoMapper.cs
+6 -2
View File
@@ -506,7 +506,9 @@ public static class SiteCommandDtoMapper
AuthType = artifact.AuthType,
AuthConfiguration = OrEmpty(artifact.AuthConfiguration),
MethodDefinitionsJson = OrEmpty(artifact.MethodDefinitionsJson),
TimeoutSeconds = artifact.TimeoutSeconds
TimeoutSeconds = artifact.TimeoutSeconds,
MaxRetries = artifact.MaxRetries,
RetryDelay = Duration.FromTimeSpan(artifact.RetryDelay)
};
}
@@ -523,7 +525,9 @@ public static class SiteCommandDtoMapper
dto.AuthType,
NullIfEmpty(dto.AuthConfiguration),
NullIfEmpty(dto.MethodDefinitionsJson),
dto.TimeoutSeconds);
dto.TimeoutSeconds,
dto.MaxRetries,
dto.RetryDelay?.ToTimeSpan() ?? TimeSpan.Zero);
}
/// <summary>Projects a <see cref="DatabaseConnectionArtifact"/> onto the wire.</summary>
src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcServer.cs
+4 -4
View File
@@ -467,12 +467,12 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
// Telemetry follow-on: the connection is now fully established (Subscribe
// succeeded, so no leak via the catch above). Count it up here and balance
// it in the finally below so the scadabridge.site.connection.up gauge is
// decremented on EVERY exit path — normal completion, client-cancel /
// it in the finally below so the scadabridge.site.stream.subscribers.active
// gauge is decremented on EVERY exit path — normal completion, client-cancel /
// duplicate-replacement (OperationCanceledException), server shutdown
// (CancelAllStreams -> Cts.Cancel), and any other exception — guaranteeing
// exactly one Closed per Opened and a gauge that never drifts up.
ScadaBridgeTelemetry.SiteConnectionOpened();
ScadaBridgeTelemetry.SiteStreamSubscriberOpened();
try
{
// R2 — event batching. The pump replaces the old per-event
@@ -500,7 +500,7 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
}
finally
{
ScadaBridgeTelemetry.SiteConnectionClosed();
ScadaBridgeTelemetry.SiteStreamSubscriberClosed();
_streamSubscriber.RemoveSubscriber(relayActor);
_actorSystem!.Stop(relayActor);
channel.Writer.TryComplete();
src/ZB.MOM.WW.ScadaBridge.Communication/Protos/central_control.proto
+6
View File
@@ -228,6 +228,12 @@ message SiteHealthReportDto {
// Absent means UNKNOWN, never zero — a failed backlog read rendered as 0
// would report a broken replication pair as perfectly healthy.
google.protobuf.Int64Value local_db_oplog_backlog = 28;
// Whether this node spawned its SiteReconciliationActor at boot. Absent means
// "not reported" — a central self-report (where it is meaningless) or a site
// node on a pre-row-73 build. false means the node booted WITHOUT the startup
// self-heal path, which is a misconfiguration a healthy-looking site would
// otherwise hide entirely.
google.protobuf.BoolValue startup_reconciler_active = 29;
}
// Central -> Site: health report ack, so a lost report is observable.
src/ZB.MOM.WW.ScadaBridge.Communication/Protos/site_command.proto
+5
View File
@@ -224,6 +224,11 @@ message ExternalSystemArtifactDto {
string auth_configuration = 4; // empty string represents null
string method_definitions_json = 5; // empty string represents null
int32 timeout_seconds = 6;
// Store-and-Forward retry policy for this system's cached calls. Additive (fields 7-8);
// a pre-existing site decodes them as 0/absent, which is exactly the "unset" value that
// makes the engine defaults apply — i.e. the old behaviour.
int32 max_retries = 7;
google.protobuf.Duration retry_delay = 8;
}
message DatabaseConnectionArtifactDto {
src/ZB.MOM.WW.ScadaBridge.Communication/ServiceCollectionExtensions.cs
+26 -2
View File
@@ -42,9 +42,33 @@ public static class ServiceCollectionExtensions
services.AddSingleton<SiteAlarmLiveCacheService>();
services.AddSingleton<ISiteAlarmLiveCache>(sp => sp.GetRequiredService<SiteAlarmLiveCacheService>());
return services;
}
/// <summary>
/// Registers the communication services that only a CENTRAL node can construct.
/// Call alongside <see cref="AddCommunication"/> from the Central composition root only.
/// </summary>
/// <remarks>
/// <para>
/// <see cref="ReconcileService"/> is scoped and takes <c>IDeploymentManagerRepository</c> —
/// an EF repository registered by <c>AddConfigurationDatabase</c>, which only central calls.
/// It used to live in the shared <see cref="AddCommunication"/> on the argument that it was
/// "harmless on site hosts: only the central actor ever resolves it". That is true of the
/// runtime behaviour but not of the container: the registration was UNCONSTRUCTIBLE on a
/// site node, so <c>ValidateOnBuild</c> failed the whole site graph and the site branch
/// could not have scope validation turned on at all (deferred register row 50). A
/// registration nothing can ever build is not harmless — it is the thing that stops the
/// validator from being usable, and it hides whatever real defect the validator would
/// otherwise have found.
/// </para>
/// </remarks>
/// <param name="services">The DI service collection to register services into.</param>
/// <returns>The same <see cref="IServiceCollection"/> to allow chaining.</returns>
public static IServiceCollection AddCentralCommunication(this IServiceCollection services)
{
// Startup reconciliation handler — scoped (holds scoped repositories), resolved
// per-request by CentralCommunicationActor inside a DI scope. Harmless on site
// hosts: only the central actor ever resolves it.
// per-request by CentralCommunicationActor inside a DI scope.
services.AddScoped<ReconcileService>();
return services;
src/ZB.MOM.WW.ScadaBridge.Communication/SiteCommandGrpc/SiteCommand.cs
+518 -429
View File
File diff suppressed because it is too large Load Diff
src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/BundleUnlockAttemptEntityTypeConfiguration.cs
+46
View File
@@ -0,0 +1,46 @@
using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Metadata.Builders;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Transport;
namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Configurations;
/// <summary>
/// Maps the <see cref="BundleUnlockAttempt"/> POCO to the central
/// <c>BundleUnlockAttempt</c> table — the fleet-wide sliding window behind the Transport
/// component's per-client bundle-unlock throttle (register row 52). Operational history,
/// NOT audit, so the table is non-partitioned, standard <c>[PRIMARY]</c> filegroup, no
/// DB-role restriction.
/// </summary>
/// <remarks>
/// One named index, <c>IX_BundleUnlockAttempt_Client_Attempted</c>, serves both access
/// paths: the window count seeks on <c>ClientKey</c> and ranges on
/// <c>AttemptedAtUtc</c>, and the opportunistic prune scans the same index by time. A
/// second time-only index was deliberately not added — the table is kept small by the
/// prune, so the leading-column seek is cheap enough for the prune to share.
/// </remarks>
public class BundleUnlockAttemptEntityTypeConfiguration : IEntityTypeConfiguration<BundleUnlockAttempt>
{
/// <summary>
/// Configures the EF Core entity type mapping for <see cref="BundleUnlockAttempt"/>.
/// </summary>
/// <param name="builder">The entity type builder to configure.</param>
public void Configure(EntityTypeBuilder<BundleUnlockAttempt> builder)
{
builder.ToTable("BundleUnlockAttempt");
// Surrogate long identity assigned by the store.
builder.HasKey(a => a.Id);
// Bucket key: an IP literal or the "<unattributed>" sentinel. 64 chars covers an
// IPv6 address with a scope id; ASCII, so non-unicode (varchar) like every other
// catalog-bounded column in the schema.
builder.Property(a => a.ClientKey).HasMaxLength(64).IsUnicode(false).IsRequired();
builder.Property(a => a.AttemptedAtUtc).IsRequired();
// Window index — seeks the bucket, ranges over the trailing hour, and is
// reused by the prune's time scan. Name locked for migration discoverability.
builder.HasIndex(a => new { a.ClientKey, a.AttemptedAtUtc })
.HasDatabaseName("IX_BundleUnlockAttempt_Client_Attempted");
}
}
src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/DeploymentConfiguration.cs
+20 -1
View File
@@ -39,7 +39,26 @@ public class DeploymentRecordConfiguration : IEntityTypeConfiguration<Deployment
builder.HasIndex(d => d.DeploymentId).IsUnique();
builder.HasIndex(d => d.InstanceId);
builder.HasIndex(d => d.DeployedAt);
// Deployment-list sort key. Every list read orders DeployedAt DESC, Id DESC
// (DeploymentManagerRepository.QueryDeploymentListPageAsync and
// QueryDeploymentSummariesAsync) — Id is the deterministic tie-break, because
// a bulk site deploy stamps many rows in the same DeployedAt instant and a
// non-deterministic order would let a row repeat or vanish across pages.
// Declaring both key columns descending lets a forward scan feed the paged
// Skip/Take directly instead of the server sorting the whole filtered set per
// page (deferred-work register row 35).
//
// This REPLACES the old single-column HasIndex(d => d.DeployedAt): that index
// is strictly redundant here, not merely overlapping. DeployedAt leads both, so
// any predicate the old one served the composite serves too (direction does not
// affect seekability — SQL Server scans either way), and the leaf is the same
// width, since Id is the clustered PK and was therefore already sitting in the
// old index's leaf as the row locator. Keeping it would buy nothing and cost a
// second index maintained on every insert and status transition.
builder.HasIndex(d => new { d.DeployedAt, d.Id })
.IsDescending(true, true)
.HasDatabaseName("IX_DeploymentRecords_DeployedAt_Id");
}
}
src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260815154803_AddBundleUnlockAttemptTable.Designer.cs
Generated
+2121
View File
File diff suppressed because it is too large Load Diff
src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260815154803_AddBundleUnlockAttemptTable.cs
+41
View File
@@ -0,0 +1,41 @@
using System;
using Microsoft.EntityFrameworkCore.Migrations;
#nullable disable
namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Migrations
{
/// <inheritdoc />
public partial class AddBundleUnlockAttemptTable : Migration
{
/// <inheritdoc />
protected override void Up(MigrationBuilder migrationBuilder)
{
migrationBuilder.CreateTable(
name: "BundleUnlockAttempt",
columns: table => new
{
Id = table.Column<long>(type: "bigint", nullable: false)
.Annotation("SqlServer:Identity", "1, 1"),
ClientKey = table.Column<string>(type: "varchar(64)", unicode: false, maxLength: 64, nullable: false),
AttemptedAtUtc = table.Column<DateTime>(type: "datetime2", nullable: false)
},
constraints: table =>
{
table.PrimaryKey("PK_BundleUnlockAttempt", x => x.Id);
});
migrationBuilder.CreateIndex(
name: "IX_BundleUnlockAttempt_Client_Attempted",
table: "BundleUnlockAttempt",
columns: new[] { "ClientKey", "AttemptedAtUtc" });
}
/// <inheritdoc />
protected override void Down(MigrationBuilder migrationBuilder)
{
migrationBuilder.DropTable(
name: "BundleUnlockAttempt");
}
}
}
src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260815165452_AddDeploymentRecordListSortIndex.Designer.cs
Generated
+2123
View File
File diff suppressed because it is too large Load Diff
src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260815165452_AddDeploymentRecordListSortIndex.cs
+60
View File
@@ -0,0 +1,60 @@
using Microsoft.EntityFrameworkCore.Migrations;
#nullable disable
namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Migrations
{
/// <summary>
/// Replaces the single-column <c>IX_DeploymentRecords_DeployedAt</c> with the
/// composite <c>IX_DeploymentRecords_DeployedAt_Id (DeployedAt DESC, Id DESC)</c>,
/// which is the deployment list's exact sort key (deferred-work register row 35).
/// </summary>
/// <remarks>
/// The list read (<c>DeploymentManagerRepository.QueryDeploymentListPageAsync</c>)
/// joins <c>Instance</c> to make <c>SiteId</c> filterable, then orders
/// <c>DeployedAt DESC, Id DESC</c> and pages with Skip/Take. With both key columns
/// declared descending, a forward scan of this index already delivers rows in the
/// requested order, so the plan can feed the Top-N page directly rather than sorting
/// the whole filtered set on every page request.
///
/// The drop is safe rather than merely tolerable: <c>DeployedAt</c> leads both
/// indexes, so every predicate the old one served the composite serves as well
/// (index direction does not affect seekability — SQL Server scans either
/// direction), and the leaf rows are the same width, because <c>Id</c> is the
/// clustered primary key and was therefore already present in the old nonclustered
/// index's leaf as the row locator. What changes is only that <c>Id</c> is now a
/// declared, ordered key column instead of an implicit locator. Keeping both would
/// add a second index to maintain on every insert and status transition for no read
/// benefit. <c>SystemArtifactDeploymentRecords</c> has its own separate
/// <c>DeployedAt</c> index and is deliberately untouched.
/// </remarks>
public partial class AddDeploymentRecordListSortIndex : Migration
{
/// <inheritdoc />
protected override void Up(MigrationBuilder migrationBuilder)
{
migrationBuilder.DropIndex(
name: "IX_DeploymentRecords_DeployedAt",
table: "DeploymentRecords");
migrationBuilder.CreateIndex(
name: "IX_DeploymentRecords_DeployedAt_Id",
table: "DeploymentRecords",
columns: new[] { "DeployedAt", "Id" },
descending: new bool[0]);
}
/// <inheritdoc />
protected override void Down(MigrationBuilder migrationBuilder)
{
migrationBuilder.DropIndex(
name: "IX_DeploymentRecords_DeployedAt_Id",
table: "DeploymentRecords");
migrationBuilder.CreateIndex(
name: "IX_DeploymentRecords_DeployedAt",
table: "DeploymentRecords",
column: "DeployedAt");
}
}
}
src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/ScadaBridgeDbContextModelSnapshot.cs
+29 -2
View File
@@ -267,13 +267,15 @@ namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Migrations
b.HasKey("Id");
b.HasIndex("DeployedAt");
b.HasIndex("DeploymentId")
.IsUnique();
b.HasIndex("InstanceId");
b.HasIndex("DeployedAt", "Id")
.IsDescending()
.HasDatabaseName("IX_DeploymentRecords_DeployedAt_Id");
b.ToTable("DeploymentRecords");
});
@@ -1716,6 +1718,31 @@ namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Migrations
b.ToTable("TemplateScripts");
});
modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Transport.BundleUnlockAttempt", b =>
{
b.Property<long>("Id")
.ValueGeneratedOnAdd()
.HasColumnType("bigint");
SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property<long>("Id"));
b.Property<DateTime>("AttemptedAtUtc")
.HasColumnType("datetime2");
b.Property<string>("ClientKey")
.IsRequired()
.HasMaxLength(64)
.IsUnicode(false)
.HasColumnType("varchar(64)");
b.HasKey("Id");
b.HasIndex("ClientKey", "AttemptedAtUtc")
.HasDatabaseName("IX_BundleUnlockAttempt_Client_Attempted");
b.ToTable("BundleUnlockAttempt", (string)null);
});
modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Entities.AuditLogRow", b =>
{
b.Property<Guid>("EventId")
src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/BundleUnlockAttemptRepository.cs
+70
View File
@@ -0,0 +1,70 @@
using Microsoft.EntityFrameworkCore;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Transport;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Repositories;
/// <summary>
/// EF Core implementation of <see cref="IBundleUnlockAttemptRepository"/> over the central
/// <c>BundleUnlockAttempt</c> table (register row 52). Plain reads/writes against the
/// shared <see cref="ScadaBridgeDbContext"/>; the prune uses <c>ExecuteDeleteAsync</c> so
/// no rows are materialised into the change tracker.
/// </summary>
public class BundleUnlockAttemptRepository : IBundleUnlockAttemptRepository
{
private readonly ScadaBridgeDbContext _context;
/// <summary>
/// Initializes a new instance of the <see cref="BundleUnlockAttemptRepository"/> class.
/// </summary>
/// <param name="context">The EF Core database context.</param>
public BundleUnlockAttemptRepository(ScadaBridgeDbContext context)
{
_context = context ?? throw new ArgumentNullException(nameof(context));
}
/// <inheritdoc />
public async Task<int> CountAttemptsSinceAsync(
string clientKey, DateTime sinceUtc, CancellationToken ct = default)
{
ArgumentException.ThrowIfNullOrWhiteSpace(clientKey);
return await _context.BundleUnlockAttempts
.AsNoTracking()
.Where(a => a.ClientKey == clientKey && a.AttemptedAtUtc >= sinceUtc)
.CountAsync(ct);
}
/// <inheritdoc />
public async Task RecordAttemptAsync(
string clientKey, DateTime attemptedAtUtc, CancellationToken ct = default)
{
ArgumentException.ThrowIfNullOrWhiteSpace(clientKey);
_context.BundleUnlockAttempts.Add(new BundleUnlockAttempt
{
ClientKey = clientKey,
AttemptedAtUtc = attemptedAtUtc,
});
await _context.SaveChangesAsync(ct);
}
/// <inheritdoc />
public async Task<int> PruneAttemptsOlderThanAsync(
DateTime beforeUtc, int maxRows, CancellationToken ct = default)
{
if (maxRows < 1)
{
throw new ArgumentOutOfRangeException(nameof(maxRows), "Prune batch size must be at least 1.");
}
// Bounded by construction: OrderBy + Take caps the delete regardless of how much
// history accumulated while the prune was failing or the node was down, so an
// opportunistic sweep on a request path can never become an unbounded delete.
return await _context.BundleUnlockAttempts
.Where(a => a.AttemptedAtUtc < beforeUtc)
.OrderBy(a => a.AttemptedAtUtc)
.Take(maxRows)
.ExecuteDeleteAsync(ct);
}
}
src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/ScadaBridgeDbContext.cs
+5
View File
@@ -13,6 +13,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Schemas;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Scripts;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.SecuredWrites;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Transport;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Security;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates;
@@ -152,6 +153,10 @@ public class ScadaBridgeDbContext : DbContext, IDataProtectionKeyContext
/// <summary>Gets the set of hourly KPI rollups (pre-aggregated long-range trend backbone).</summary>
public DbSet<KpiRollupHourly> KpiRollupHourly => Set<KpiRollupHourly>();
// Transport (bundle export/import)
/// <summary>Gets the set of recorded bundle-unlock attempts — the fleet-wide sliding window behind the per-client unlock throttle (register row 52).</summary>
public DbSet<BundleUnlockAttempt> BundleUnlockAttempts => Set<BundleUnlockAttempt>();
// Data Protection Keys (for shared ASP.NET Data Protection across nodes)
/// <summary>Gets the set of data protection keys.</summary>
public DbSet<DataProtectionKey> DataProtectionKeys => Set<DataProtectionKey>();
src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/ServiceCollectionExtensions.cs
+4
View File
@@ -87,6 +87,10 @@ public static class ServiceCollectionExtensions
services.AddScoped<ISecuredWriteRepository, SecuredWriteRepository>();
services.AddScoped<ISharedSchemaRepository, SharedSchemaRepository>();
services.AddScoped<IKpiHistoryRepository, KpiHistoryRepository>();
// Register row 52: the fleet-wide bundle-unlock throttle window. Central-only in
// practice (Transport is registered in the host's Central branch), but registered
// here with every other repository so the ConfigurationDatabase surface stays uniform.
services.AddScoped<IBundleUnlockAttemptRepository, BundleUnlockAttemptRepository>();
// Auth re-arch: inbound API keys are no longer persisted in SQL Server —
// the repository now exposes only API-method access, so a plain scoped
// registration suffices (no peppered-hasher accessor to wire).
src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs
+24
View File
@@ -3,6 +3,7 @@ using Akka.Event;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Protocol;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Management;
using ZB.MOM.WW.ScadaBridge.Commons.Observability;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Alarms;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.HealthMonitoring;
@@ -312,6 +313,10 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
_log.Info("[{0}] Entering Connecting state", _connectionName);
_healthCollector.UpdateConnectionHealth(_connectionName, ConnectionHealth.Connecting);
_healthCollector.UpdateConnectionEndpoint(_connectionName, "Connecting");
// Register row 54 — mirror the state into the data-connection gauges. Kept alongside
// the health-collector write (rather than derived from it) so the metric never
// depends on the health-report path being alive.
ScadaBridgeTelemetry.RecordDataConnectionState(_connectionName, ConnectionHealth.Connecting);
Become(Connecting);
Self.Tell(new AttemptConnect());
}
@@ -394,6 +399,8 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
_healthCollector.UpdateTagResolution(_connectionName, TotalSubscribedTagCount, ResolvedTagCount);
var endpointLabel = _backupConfig == null ? "Connected" : $"Connected to {_activeEndpoint.ToString().ToLower()}";
_healthCollector.UpdateConnectionEndpoint(_connectionName, endpointLabel);
// Register row 54 — the only transition that raises the connected gauge.
ScadaBridgeTelemetry.RecordDataConnectionState(_connectionName, ConnectionHealth.Connected);
Become(Connected);
Stash.UnstashAll();
}
@@ -475,6 +482,9 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
_log.Warning("[{0}] Entering Reconnecting state", _connectionName);
_healthCollector.UpdateConnectionHealth(_connectionName, ConnectionHealth.Disconnected);
_healthCollector.UpdateConnectionEndpoint(_connectionName, "Disconnected");
// Register row 54 — drops this connection out of the connected gauge while the
// total holds, which is exactly the connected < total alert condition.
ScadaBridgeTelemetry.RecordDataConnectionState(_connectionName, ConnectionHealth.Disconnected);
// Track unstable connections toward failover.
// If we were connected for less than the stability threshold, this counts
@@ -1275,6 +1285,20 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
// Fire-and-forget release of every handle this request created redundantly.
_ = UnsubscribeIdsAsync(_adapter, redundantIds);
// Publish the resolution counts this request just changed. Without this push a
// connection whose tags ALL resolve on the first attempt reported 0/0 forever:
// the only other pushes are BecomeConnected's (which fires before any instance
// has subscribed, so its 0/0 is legitimate but is never superseded),
// HandleBatchSubscribeCompleted's (reconnect re-subscribe / resolution probe) and
// HandleUnsubscribe's (teardown) — and ScheduleTagResolutionRetry below cancels
// the retry timer outright when nothing is unresolved, so no later tick pushed
// either. The blind spot was therefore worst on a perfectly healthy connection,
// which is precisely when the "device still booting / tag path unresolved"
// diagnostic is supposed to read clean rather than empty. Mirrors the push site
// in HandleBatchSubscribeCompleted, and stays on the actor thread where both
// derived counts are stable.
_healthCollector.UpdateTagResolution(_connectionName, TotalSubscribedTagCount, ResolvedTagCount);
// Now that every tag is registered in
// _subscriptionsByInstance, deliver the values captured by the initial read.
// Re-entering via Self reuses HandleTagValueReceived's generation guard, fan-out
src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionManagerActor.cs
+5
View File
@@ -4,6 +4,7 @@ using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Protocol;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Management;
using ZB.MOM.WW.ScadaBridge.Commons.Observability;
using ZB.MOM.WW.ScadaBridge.Commons.Serialization;
using ZB.MOM.WW.ScadaBridge.Commons.Types.DataConnections;
using ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Adapters;
@@ -307,6 +308,10 @@ public class DataConnectionManagerActor : ReceiveActor
Context.Stop(actor);
_connectionActors.Remove(command.ConnectionName);
_healthCollector.RemoveConnection(command.ConnectionName);
// Register row 54 — drop it from the gauges too, so an undeployed connection
// stops counting toward the total instead of lingering as a permanently
// not-connected row that would hold the connected < total alert on forever.
ScadaBridgeTelemetry.RemoveDataConnection(command.ConnectionName);
_log.Info("Removed DataConnectionActor for {0}", command.ConnectionName);
}
}
src/ZB.MOM.WW.ScadaBridge.DeploymentManager/ArtifactDeploymentService.cs
+22 -2
View File
@@ -192,8 +192,13 @@ public class ArtifactDeploymentService
m.ReturnDefinition
}))
: null;
// MaxRetries/RetryDelay ride along so the site's Store-and-Forward buffering honours
// the centrally-configured per-system retry policy. Before they were carried, a site
// silently fell back to the engine defaults (50 x 30 s) for every cached call, whatever
// central had configured — the DatabaseConnectionArtifact below always carried them.
externalSystemArtifacts.Add(new ExternalSystemArtifact(
es.Name, es.EndpointUrl, es.AuthType, es.AuthConfiguration, methodsJson, es.TimeoutSeconds));
es.Name, es.EndpointUrl, es.AuthType, es.AuthConfiguration, methodsJson, es.TimeoutSeconds,
es.MaxRetries, es.RetryDelay));
}
// Map database connections
@@ -286,7 +291,22 @@ public class ArtifactDeploymentService
site.Id, globals, cancellationToken, deploymentId);
}
// Deploy to each site in parallel with per-site timeout
// Deploy to each site in parallel with per-site timeout.
//
// Deferred-work register row 60 — what makes this fan-out safe, stated precisely
// because the invariant is easy to break from a distance:
// * `globals` is NOT EF state. FetchGlobalArtifactsAsync projects the entities
// into pure Commons message DTOs and drops the entities, so the snapshot the
// lambda closes over cannot touch a DbContext at all.
// * `site` IS a tracked EF entity, and it is the one piece of EF state that
// genuinely crosses into the parallel phase. Only already-materialised scalar
// properties are read below (Id, SiteIdentifier, Name) and NO navigation
// property is traversed — with lazy loading off (pinned by
// ConfigurationDatabase.Tests.AddConfigurationDatabase_DoesNotEnableLazyLoading)
// those are plain field reads that never re-enter the shared context.
// So: do not traverse a navigation property on `site` in here, and do not move
// per-site command building (which DOES query) inside this lambda — either one
// puts concurrent load on one non-thread-safe DbContext.
var tasks = sites.Select(async site =>
{
try
src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthAggregator.cs
+44 -2
View File
@@ -50,7 +50,8 @@ public class CentralHealthAggregator : BackgroundService, ICentralHealthAggregat
FirstSeenAt = now,
LastSequenceNumber = report.SequenceNumber,
IsOnline = true,
IsMetricsStale = false
IsMetricsStale = false,
LocalDbBacklogAboveSince = NextBacklogAboveSince(null, report, now)
// LastStatusChangeAt stays null: first observation is not a flip.
};
@@ -84,7 +85,9 @@ public class CentralHealthAggregator : BackgroundService, ICentralHealthAggregat
IsMetricsStale = false,
// Stamp a flip only when this report brings the site back online;
// a report from an already-online site leaves the timestamp be.
LastStatusChangeAt = existing.IsOnline ? existing.LastStatusChangeAt : now
LastStatusChangeAt = existing.IsOnline ? existing.LastStatusChangeAt : now,
LocalDbBacklogAboveSince =
NextBacklogAboveSince(existing.LocalDbBacklogAboveSince, report, now)
};
if (_siteStates.TryUpdate(report.SiteId, updated, existing))
@@ -320,4 +323,43 @@ public class CentralHealthAggregator : BackgroundService, ICentralHealthAggregat
}
}
}
/// <summary>
/// Advances the sustained-backlog anchor for one incoming report (register row 33).
/// </summary>
/// <remarks>
/// Three cases, and the null one is the subtle one:
/// <list type="bullet">
/// <item><description>
/// Backlog <c>null</c> — no reading. An unreplicated site (rig site-b/site-c) reports this
/// forever, and a replicated site reports it when a poll fails. Carry the previous anchor
/// through UNCHANGED: null is not zero, so it is neither evidence of a drain (which would
/// justify clearing) nor of a backlog (which would justify starting one). An unreplicated
/// site therefore never accumulates an anchor and can never be badged.
/// </description></item>
/// <item><description>
/// Backlog above the threshold — start the clock if it is not already running. Keeping the
/// ORIGINAL start instant is what makes the window sustained rather than sliding.
/// </description></item>
/// <item><description>
/// Backlog at or below the threshold — clear. This is the daily purge burst's exit path:
/// the spike drains within a few reports and the anchor is discarded long before the dwell
/// elapses, so the expected burst never badges the site.
/// </description></item>
/// </list>
/// </remarks>
/// <param name="current">The anchor currently held for this site, if any.</param>
/// <param name="report">The incoming health report.</param>
/// <param name="now">The current time.</param>
/// <returns>The anchor to store.</returns>
private DateTimeOffset? NextBacklogAboveSince(
DateTimeOffset? current, SiteHealthReport report, DateTimeOffset now)
{
if (report.LocalDbOplogBacklog is not { } backlog)
{
return current;
}
return backlog > _options.LocalDbBacklogThreshold ? current ?? now : null;
}
}
src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/HealthMonitoringOptions.cs
+28
View File
@@ -32,4 +32,32 @@ public class HealthMonitoringOptions
/// (~4 missed 30s reports).
/// </summary>
public TimeSpan MetricsStaleTimeout { get; set; } = TimeSpan.FromMinutes(2);
/// <summary>
/// Unacked LocalDb replication oplog depth above which a site is considered backlogged
/// (register row 33). Crossing this alone shows nothing — the backlog must also stay
/// above it for <see cref="LocalDbBacklogSustainedFor"/> before the Health dashboard
/// badges the site. Must be positive. Default: <c>1000</c> entries.
/// </summary>
public long LocalDbBacklogThreshold { get; set; } = 1000;
/// <summary>
/// How long the oplog backlog must stay continuously above
/// <see cref="LocalDbBacklogThreshold"/> before the Health dashboard badges the site
/// (register row 33).
/// </summary>
/// <remarks>
/// This dwell time is the whole point of the signal, not a refinement of it. A bare
/// threshold would fire every single day on a healthy replicated site: the `site_events`
/// retention purge and the storage-cap trim are CDC-captured like any other write, so the
/// backlog jumps by the deleted batch size and then drains as the peer acks it. That spike
/// is expected and documented (see <c>docs/deployment/topology-guide.md</c> → *Reading the
/// replication backlog*). Healthy means the backlog RETURNS TO ~0; unhealthy means it
/// stays up. Only a sustained window can tell those apart, so the default is set well
/// clear of a purge drain. Must be positive and no shorter than
/// <see cref="ReportInterval"/> — a window shorter than one report interval could be
/// satisfied by a single sample, which is exactly the bare threshold this exists to avoid.
/// Default: 15 minutes.
/// </remarks>
public TimeSpan LocalDbBacklogSustainedFor { get; set; } = TimeSpan.FromMinutes(15);
}
src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/HealthMonitoringOptionsValidator.cs
+24
View File
@@ -64,5 +64,29 @@ public sealed class HealthMonitoringOptionsValidator : OptionsValidatorBase<Heal
$"must be >= ReportInterval ({options.ReportInterval}): a stale window shorter than one " +
"report interval would flag every healthy site, since reports cannot arrive faster than " +
"they are emitted.");
builder.RequireThat(options.LocalDbBacklogThreshold > 0,
$"ScadaBridge:HealthMonitoring:LocalDbBacklogThreshold must be positive " +
$"(was {options.LocalDbBacklogThreshold}); a zero/negative threshold would badge every " +
"replicated site permanently, since an idle backlog of 0 already sits at or above it.");
builder.RequireThat(options.LocalDbBacklogSustainedFor > TimeSpan.Zero,
$"ScadaBridge:HealthMonitoring:LocalDbBacklogSustainedFor must be a positive duration " +
$"(was {options.LocalDbBacklogSustainedFor}); it is the dwell time a backlog must stay " +
"above the threshold before the site is badged.");
// A dwell shorter than one report interval can be satisfied by a single sample,
// degrading the sustained check back into the bare threshold it exists to replace —
// which would fire on every daily purge burst. Same De Morgan guard shape as the
// pairs above: silent unless BOTH fields are positive and the dwell is too short,
// so the dedicated positive-value checks own that failure.
builder.RequireThat(
!(options.ReportInterval > TimeSpan.Zero
&& options.LocalDbBacklogSustainedFor > TimeSpan.Zero
&& options.LocalDbBacklogSustainedFor < options.ReportInterval),
$"ScadaBridge:HealthMonitoring:LocalDbBacklogSustainedFor ({options.LocalDbBacklogSustainedFor}) " +
$"must be >= ReportInterval ({options.ReportInterval}): a dwell shorter than one report " +
"interval can be satisfied by a single sample, which turns the sustained check back into " +
"a bare threshold and re-introduces the daily purge-burst false positive it exists to avoid.");
}
}
src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ISiteHealthCollector.cs
+29 -5
View File
@@ -162,14 +162,38 @@ public interface ISiteHealthCollector
/// service. Point-in-time: values are NOT reset on <see cref="CollectReport"/>.
/// </summary>
/// <param name="connected">
/// Whether a sync session is currently running. <see langword="false"/> is the
/// normal state on a node with no peer configured — replication ships default-OFF.
/// Whether a sync session is currently running, or <see langword="null"/> when there
/// is no reading to give — which is the state of a node with replication UNCONFIGURED
/// (register row 80). <see langword="false"/> means "replication is configured on this
/// node and the peer session is currently down", i.e. a genuine fault worth a badge;
/// reporting that on a deliberately-unreplicated site reads as a fault that isn't one.
/// </param>
/// <param name="oplogBacklog">
/// Unacked oplog entries, or <see langword="null"/> when unknown. Pass null through
/// unchanged: a failed poll rendered as 0 would report a broken pair as healthy.
/// Unacked oplog entries, or <see langword="null"/> when unknown (failed poll) or not
/// applicable (replication unconfigured). Pass null through unchanged: a failed poll
/// rendered as 0 would report a broken pair as healthy.
/// </param>
void SetLocalDbReplicationStatus(bool connected, long? oplogBacklog)
void SetLocalDbReplicationStatus(bool? connected, long? oplogBacklog)
{
// Default no-op so test fakes do not need to be updated.
}
/// <summary>
/// Record whether this node spawned its <c>SiteReconciliationActor</c> at boot, for the
/// <c>StartupReconcilerActive</c> field of every subsequent <see cref="CollectReport"/>
/// (deferred-work register row 73).
/// </summary>
/// <remarks>
/// Called ONCE at boot by the Host, which owns the spawn decision — this is a latched
/// boot fact, not a polled gauge, so there is no reporter hosted service behind it.
/// A collector that is never told (central's self-report, or any test host) leaves the
/// field null, which reads as "not reported" rather than "reconciler absent".
/// </remarks>
/// <param name="active">
/// <see langword="true"/> when the reconciler was created; <see langword="false"/> when
/// it was skipped because no <c>IDeploymentConfigFetcher</c> was available.
/// </param>
void SetStartupReconcilerActive(bool active)
{
// Default no-op so test fakes do not need to be updated.
}
src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/Kpi/SiteHealthKpiSampleSource.cs
+28
View File
@@ -49,6 +49,8 @@ public sealed class SiteHealthKpiSampleSource : IKpiSampleSource
private const string MetricDisabledInstances = "disabledInstances";
private const string MetricAuditBacklogPending = "auditBacklogPending";
private const string MetricEventLogWriteFailures = KpiMetrics.SiteHealth.EventLogWriteFailures;
private const string MetricLocalDbOplogBacklog = KpiMetrics.SiteHealth.LocalDbOplogBacklog;
private const string MetricLocalDbReplicationConnected = KpiMetrics.SiteHealth.LocalDbReplicationConnected;
private readonly ICentralHealthAggregator _aggregator;
@@ -136,6 +138,32 @@ public sealed class SiteHealthKpiSampleSource : IKpiSampleSource
report.SiteAuditBacklog?.PendingCount ?? 0));
samples.Add(Sample(
capturedAtUtc, MetricEventLogWriteFailures, siteId, report.SiteEventLogWriteFailures));
// Register row 33: LocalDb replication. Both fields are NULLABLE and null is
// load-bearing — it means "no reading" (replication not configured, as on rig
// site-b/site-c, or a failed poll), NOT zero. So unlike the audit-backlog line
// above, these are CONDITIONALLY emitted rather than coerced with `?? 0`:
//
// - A fabricated 0 backlog would render a pair that cannot read its own oplog,
// or one that simply isn't replicated, as perfectly healthy — and would do it
// once a minute, forever, burying the real series for the sites that are.
// - For the connected flag, `false` and null are genuinely different states: a
// configured link that is DOWN versus no link at all. Collapsing null to 0
// would alarm on every unreplicated site permanently.
//
// The cost is that a site's sample count varies (12 unreplicated, 14 replicated).
// That is the intended shape: a GAP in these two series means "not replicated / not
// reporting", which is a truthful reading, whereas a flat 0 line is a lie.
if (report.LocalDbOplogBacklog is { } backlog)
{
samples.Add(Sample(capturedAtUtc, MetricLocalDbOplogBacklog, siteId, backlog));
}
if (report.LocalDbReplicationConnected is { } connected)
{
samples.Add(Sample(
capturedAtUtc, MetricLocalDbReplicationConnected, siteId, connected ? 1 : 0));
}
}
private static KpiSample Sample(
src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/LocalDbReplicationStatusReporter.cs
+34
View File
@@ -22,6 +22,23 @@ namespace ZB.MOM.WW.ScadaBridge.HealthMonitoring;
/// report a replication pair that cannot read its own oplog as perfectly healthy.
/// </para>
/// <para>
/// <b>An UNCONFIGURED node reports null for both fields</b> (register row 80). The
/// replication engine is registered unconditionally and sits inert without a peer, so
/// <c>ISyncStatus</c> answers <c>Connected=false</c> / <c>OplogBacklog=0</c> on a site
/// that was never meant to replicate at all. Publishing that pair is actively misleading:
/// a dashboard reader sees "connected: false" on a healthy unreplicated site and reads a
/// fault, and the central null-branch machinery (conditional KPI emission, sustained-badge
/// anchor exclusion) never receives the null it was built for. The
/// <c>replicationConfiguredProvider</c> gate therefore short-circuits the probe and pushes
/// <c>(null, null)</c> — "no reading", per the documented nullable semantics.
/// </para>
/// <para>
/// <b>The gate is supplied by the Host, not read from configuration here</b>, mirroring
/// <c>SiteEventLogReplicationCheck</c>: the Host owns
/// <c>SiteLocalDbSetup.ReplicationIsConfigured</c> (the PeerAddress-OR-ApiKey rule) and
/// HealthMonitoring must not learn to read LocalDb configuration itself.
/// </para>
/// <para>
/// <b>Cadence.</b> 30 s, matching <see cref="SiteEventLogFailureCountReporter"/> and
/// <c>SiteAuditBacklogReporter</c>. Any exception during a probe is logged and swallowed;
/// the next tick retries.
@@ -34,6 +51,7 @@ public sealed class LocalDbReplicationStatusReporter : IHostedService, IDisposab
private readonly Func<bool> _connectedProvider;
private readonly Func<long?> _oplogBacklogProvider;
private readonly Func<bool> _replicationConfiguredProvider;
private readonly ISiteHealthCollector _collector;
private readonly ILogger<LocalDbReplicationStatusReporter> _logger;
private readonly TimeSpan _refreshInterval;
@@ -43,18 +61,26 @@ public sealed class LocalDbReplicationStatusReporter : IHostedService, IDisposab
/// <summary>Initializes a new instance of <see cref="LocalDbReplicationStatusReporter"/>.</summary>
/// <param name="connectedProvider">Reads whether a peer sync session is currently running.</param>
/// <param name="oplogBacklogProvider">Reads the unacked oplog backlog, or null when unknown.</param>
/// <param name="replicationConfiguredProvider">
/// Reads whether replication is configured on this node at all (the Host's
/// <c>SiteLocalDbSetup.ReplicationIsConfigured</c>). When it answers false, both fields
/// are published as null rather than as the engine's inert false/0 defaults.
/// </param>
/// <param name="collector">The site health collector receiving the snapshot.</param>
/// <param name="logger">Logger instance.</param>
/// <param name="refreshInterval">Poll interval override; defaults to 30 s.</param>
public LocalDbReplicationStatusReporter(
Func<bool> connectedProvider,
Func<long?> oplogBacklogProvider,
Func<bool> replicationConfiguredProvider,
ISiteHealthCollector collector,
ILogger<LocalDbReplicationStatusReporter> logger,
TimeSpan? refreshInterval = null)
{
_connectedProvider = connectedProvider ?? throw new ArgumentNullException(nameof(connectedProvider));
_oplogBacklogProvider = oplogBacklogProvider ?? throw new ArgumentNullException(nameof(oplogBacklogProvider));
_replicationConfiguredProvider = replicationConfiguredProvider
?? throw new ArgumentNullException(nameof(replicationConfiguredProvider));
_collector = collector ?? throw new ArgumentNullException(nameof(collector));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
_refreshInterval = refreshInterval ?? DefaultRefreshInterval;
@@ -114,6 +140,14 @@ public sealed class LocalDbReplicationStatusReporter : IHostedService, IDisposab
{
try
{
if (!_replicationConfiguredProvider())
{
// Replication unconfigured: "no reading", NOT "disconnected with an empty
// backlog". Publishing the inert engine's false/0 here is register row 80.
_collector.SetLocalDbReplicationStatus(null, null);
return;
}
_collector.SetLocalDbReplicationStatus(
_connectedProvider(), _oplogBacklogProvider());
}
src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ServiceCollectionExtensions.cs
+16 -1
View File
@@ -170,19 +170,33 @@ public static class ServiceCollectionExtensions
/// a reference on the replication library, matching
/// <see cref="AddSiteEventLogHealthMetricsBridge"/>. Pass the backlog through as
/// nullable — coercing null to 0 would report an unreadable oplog as a healthy one.
/// <para>
/// <paramref name="replicationConfiguredProvider"/> is the register-row-80 gate: the
/// replication engine is registered unconditionally and answers <c>false</c>/<c>0</c>
/// while inert, so without it a deliberately-unreplicated site publishes
/// "disconnected, empty backlog" instead of the documented "no reading" null. The
/// caller supplies it — the Host owns <c>SiteLocalDbSetup.ReplicationIsConfigured</c>,
/// exactly as it supplies <c>SiteEventLogReplicationCheck</c> to SiteEventLogging.
/// </para>
/// </remarks>
/// <param name="services">The service collection to register into.</param>
/// <param name="connectedProvider">Given the root provider, returns a reader for "a sync session is running".</param>
/// <param name="oplogBacklogProvider">Given the root provider, returns a reader for the unacked backlog (null = unknown).</param>
/// <param name="replicationConfiguredProvider">
/// Given the root provider, returns a reader for "replication is configured on this node".
/// False makes the reporter publish null for BOTH fields.
/// </param>
/// <returns>The same <see cref="IServiceCollection"/> for chaining.</returns>
public static IServiceCollection AddLocalDbReplicationHealthBridge(
this IServiceCollection services,
Func<IServiceProvider, Func<bool>> connectedProvider,
Func<IServiceProvider, Func<long?>> oplogBacklogProvider)
Func<IServiceProvider, Func<long?>> oplogBacklogProvider,
Func<IServiceProvider, Func<bool>> replicationConfiguredProvider)
{
ArgumentNullException.ThrowIfNull(services);
ArgumentNullException.ThrowIfNull(connectedProvider);
ArgumentNullException.ThrowIfNull(oplogBacklogProvider);
ArgumentNullException.ThrowIfNull(replicationConfiguredProvider);
if (services.Any(d => d.ServiceType == typeof(LocalDbReplicationHealthBridgeMarker)))
{
@@ -193,6 +207,7 @@ public static class ServiceCollectionExtensions
services.AddHostedService(sp => new LocalDbReplicationStatusReporter(
connectedProvider(sp),
oplogBacklogProvider(sp),
replicationConfiguredProvider(sp),
sp.GetRequiredService<ISiteHealthCollector>(),
sp.GetRequiredService<ILogger<LocalDbReplicationStatusReporter>>()));
src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthCollector.cs
+24 -4
View File
@@ -20,8 +20,18 @@ public class SiteHealthCollector : ISiteHealthCollector
private long _siteEventLogWriteFailures;
// One volatile tuple rather than two independent fields: a torn read that paired a
// fresh Connected with a stale backlog would be indistinguishable from a real state.
// Null = the reporter has not yet run (or replication is not wired).
private volatile Tuple<bool, long?>? _localDbReplicationStatus;
// Null = the reporter has not yet run. Once it HAS run, an unreplicated node latches
// (null, null) — "no reading" — rather than the inert engine's (false, 0); see
// LocalDbReplicationStatusReporter and register row 80.
private volatile Tuple<bool?, long?>? _localDbReplicationStatus;
// Latched once at boot by the Host (register row 73). A tri-state int rather than a
// bool? because `volatile` cannot be applied to Nullable<bool> (a struct) and this is
// written on the startup thread but read on the report-sender thread:
// 0 = never told -> report null ("not reported": central self-report, or a test host)
// 1 = reconciler spawned
// -1 = reconciler SKIPPED (no IDeploymentConfigFetcher)
// Null must stay distinct from -1: "we don't know" is not "self-heal is missing".
private volatile int _startupReconcilerActive;
private readonly ConcurrentDictionary<string, ConnectionHealth> _connectionStatuses = new();
private readonly ConcurrentDictionary<string, TagResolutionStatus> _tagResolutionCounts = new();
private readonly ConcurrentDictionary<string, string> _connectionEndpoints = new();
@@ -109,11 +119,17 @@ public class SiteHealthCollector : ISiteHealthCollector
}
/// <inheritdoc />
public void SetLocalDbReplicationStatus(bool connected, long? oplogBacklog)
public void SetLocalDbReplicationStatus(bool? connected, long? oplogBacklog)
{
_localDbReplicationStatus = Tuple.Create(connected, oplogBacklog);
}
/// <inheritdoc />
public void SetStartupReconcilerActive(bool active)
{
_startupReconcilerActive = active ? 1 : -1;
}
/// <inheritdoc />
public void UpdateConnectionHealth(string connectionName, ConnectionHealth health)
{
@@ -256,6 +272,7 @@ public class SiteHealthCollector : ISiteHealthCollector
// Single read of the volatile tuple — two reads could straddle a reporter tick and
// pair a fresh Connected with a stale backlog.
var localDbReplication = _localDbReplicationStatus;
var startupReconciler = _startupReconcilerActive;
// Snapshot current connection and tag resolution state
var connectionStatuses = new Dictionary<string, ConnectionHealth>(_connectionStatuses);
@@ -307,7 +324,10 @@ public class SiteHealthCollector : ISiteHealthCollector
// with an empty backlog".
LocalDbReplicationConnected = localDbReplication?.Item1,
LocalDbOplogBacklog = localDbReplication?.Item2,
SiteStreamAlarmDropCount = Interlocked.Read(ref _siteStreamAlarmDropCount)
SiteStreamAlarmDropCount = Interlocked.Read(ref _siteStreamAlarmDropCount),
// Tri-state: 0 stays null ("not reported"), which must never collapse into
// false ("self-heal is missing"). Register row 73.
StartupReconcilerActive = startupReconciler == 0 ? null : startupReconciler > 0
};
}
}
src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthState.cs
+22
View File
@@ -67,6 +67,28 @@ public sealed record SiteHealthState
/// </summary>
public bool IsMetricsStale { get; init; }
/// <summary>
/// Gets the instant this site's LocalDb replication oplog backlog first rose above
/// <see cref="HealthMonitoringOptions.LocalDbBacklogThreshold"/> and stayed there, or
/// <c>null</c> if the backlog is not currently above it (register row 33).
/// </summary>
/// <remarks>
/// The anchor for the *sustained* backlog badge: the badge shows only once
/// <see cref="HealthMonitoringOptions.LocalDbBacklogSustainedFor"/> has elapsed since this
/// instant. Holding a start time rather than a boolean is what makes the signal survive the
/// expected daily purge burst — that spike raises the backlog for a drain's worth of
/// reports and then clears this back to null, never reaching the dwell.
/// <para>
/// Null is set two different ways and they mean different things. A report whose backlog is
/// at or below the threshold CLEARS it (the backlog demonstrably drained). A report whose
/// backlog is <c>null</c> — an unreplicated site, or a failed poll — LEAVES IT UNCHANGED:
/// null means "no reading", never zero, so a node that stops reporting mid-climb must not
/// silently reset the clock, and an unreplicated site (which reports null forever) never
/// starts one.
/// </para>
/// </remarks>
public DateTimeOffset? LocalDbBacklogAboveSince { get; init; }
/// <summary>
/// Gets the instant of the last online↔offline flip, or <c>null</c> if the
/// site has never changed status since it was first observed. Answers "when
src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs
+53 -4
View File
@@ -1024,6 +1024,19 @@ akka {{
// guarded-writes it (orphans are logged, never deleted). Requires the HTTP
// config fetcher; if it is somehow absent the self-heal is skipped (best-effort —
// replication remains the primary path and the next restart retries).
// The dmProxy is passed so the pass heals the RUNTIME too (register row 67): each
// enabled instance it restores gets an EnsureInstanceStartedCommand routed to the
// singleton, which starts the Instance Actor if the boot-time load missed it. Going
// through the PROXY (not a local ActorOf) keeps Instance Actors under the singleton
// regardless of which node's reconcile pass found the gap.
//
// Whichever branch is taken is LATCHED onto the site health report as
// StartupReconcilerActive (deferred-work register row 73). Before that, the skip
// was a single boot Warning and nothing else: a node that silently lost its entire
// self-heal path went on reporting itself perfectly healthy forever, and the only
// evidence was one line scrolled off the top of a log. Central badges the false
// case at Warning rather than failing the site — replication is still the primary
// path, so the node is degraded, not down.
if (deploymentConfigFetcher != null)
{
var reconcileLogger = _serviceProvider.GetRequiredService<ILoggerFactory>()
@@ -1037,17 +1050,53 @@ akka {{
_nodeOptions.NodeName,
reconcileLogger,
null,
null)),
null,
dmProxy)),
"site-reconciliation");
siteHealthCollector?.SetStartupReconcilerActive(true);
_logger.LogInformation(
"SiteReconciliationActor created (per-node startup self-heal) for site {SiteId} node {Node}",
_nodeOptions.SiteId, _nodeOptions.NodeName);
}
else
{
_logger.LogWarning(
"No IDeploymentConfigFetcher available; SiteReconciliationActor not created — "
+ "startup self-heal disabled (replication remains the primary path)");
siteHealthCollector?.SetStartupReconcilerActive(false);
// Raised from Warning to Error (register row 73). This is a misconfiguration,
// not a degraded-but-expected state: no configuration makes the fetcher
// legitimately absent on a site node, and the consequence is that neither the
// file heal nor the row-67 runtime heal will ever run on this node.
_logger.LogError(
"No IDeploymentConfigFetcher available; SiteReconciliationActor not created for "
+ "site {SiteId} node {Node} — startup self-heal is DISABLED (replication "
+ "remains the primary path). Surfaced centrally as StartupReconcilerActive=false.",
_nodeOptions.SiteId, _nodeOptions.NodeName);
// Also record it where an operator reviewing the site after the fact will find
// it — the boot log line is gone by then, but the site event log is queryable
// from the Central UI. Best-effort: a failed event write must never keep the
// node from finishing startup, which is why this is fire-and-forget with an
// observed continuation rather than an await.
var bootEventLogger = _serviceProvider
.GetService<ZB.MOM.WW.ScadaBridge.SiteEventLogging.ISiteEventLogger>();
_ = bootEventLogger?.LogEventAsync(
"deployment",
"Error",
null,
"AkkaHostedService",
"Startup self-heal disabled: SiteReconciliationActor was not created "
+ "because no IDeploymentConfigFetcher is registered.",
$"Node {_nodeOptions.NodeName} of site {_nodeOptions.SiteId} cannot "
+ "reconcile its deployed inventory at boot. Config deploys still reach "
+ "this node via LocalDb replication when it is up; a deploy that lands "
+ "while it is DOWN will not be recovered on restart.")
.ContinueWith(
t => _logger.LogWarning(
t.Exception,
"Failed to write the startup-self-heal-disabled site event."),
CancellationToken.None,
TaskContinuationOptions.OnlyOnFaulted,
TaskScheduler.Default);
}
// Audit Log — site-side telemetry actor that drains the SQLite

Some files were not shown because too many files have changed in this diff Show More