docs(register+decision): rows 69/70/78/79/80 — site-a LocalDb corruption recovered, auto-down deprecation drill-verified benign (explicit provider-class wins; re-drill on every Akka bump), replication-seeded configs start no actors (row-67 side door), OrbStack virtiofs hazard, non-null localDb health fields on unreplicated sites

CLAUDE.md

@@ -175,7 +175,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
 - Two-person MxGateway secured writes (M7): two new global roles — `Operator` (initiates) + `Verifier` (approves) — added alongside the canonical `Administrator`/`Designer`/`Deployer`/`Viewer`, with `RequireOperator`/`RequireVerifier` policies. An Operator submits a secured write from the Central UI Secured Writes page (`/operations/secured-writes`); it stays a `Pending` `PendingSecuredWrite` row until a *distinct* Verifier approves it (no-self-approval enforced server-side in the ManagementActor, plus a compare-and-swap race guard). Approval relays a `WriteTagRequest` to the site MxGateway; MxGateway-protocol connections only; each lifecycle event (submit/approve/reject/execute) emits a best-effort `AuditChannel.SecuredWrite` / `AuditKind.SecuredWrite*` central-direct-write row sharing the row id as `CorrelationId`. (SecuredWrite audit rows stamp `SourceNode` via `ICentralAuditWriter`/`INodeIdentityProvider`.) Pending secured writes expire server-side after a configurable TTL (`ManagementServiceOptions.SecuredWritePendingTtl`, default 24 h): an overdue `Pending` row is CAS'd to `Expired` (never relayed) — enforced at approve/reject and swept opportunistically on list (arch-review S2, `AuditKind.SecuredWriteExpire`).
 
 ### Cluster & Failover
-- **`auto-down` downing strategy (decision 2026-07-21 — availability over partition-safety).** Akka's `AutoDowning` provider, `auto-down-unreachable-after` = 15s: the leader among the REACHABLE members downs the unreachable peer, so a hard crash of EITHER node (active/oldest included) fails over to the survivor in ~25s. Accepted trade: a real partition → dual-active until an operator restarts one side. `keep-oldest` remains a supported `SplitBrainResolverStrategy` value (partition-safe, but an oldest-crash is a total outage — Akka's `down-if-alone` only rescues a side with ≥2 members, proven live + in 1.5.62 source). Decision record: `docs/plans/2026-07-21-auto-down-availability-decision.md`.
+- **`auto-down` downing strategy (decision 2026-07-21 — availability over partition-safety).** Akka's `AutoDowning` provider, `auto-down-unreachable-after` = 15s: the leader among the REACHABLE members downs the unreachable peer, so a hard crash of EITHER node (active/oldest included) fails over to the survivor in ~25s. Accepted trade: a real partition → dual-active until an operator restarts one side. `keep-oldest` remains a supported `SplitBrainResolverStrategy` value (partition-safe, but an oldest-crash is a total outage — Akka's `down-if-alone` only rescues a side with ≥2 members, proven live + in 1.5.62 source). Decision record: `docs/plans/2026-07-21-auto-down-availability-decision.md`. **Ignore the boot-time Akka warning claiming "keep-majority will be used instead"** — drill-verified benign (2026-08-15, decision-doc postscript): the substitution does not happen with the explicit `AutoDowning` provider-class `BuildHocon` sets. But `AutoDowning` IS deprecated, so **every Akka.NET version bump must re-run `DRILL_MODE=active bash docker/failover-drill.sh`** (register row 70).
 - Both nodes are seed nodes. `min-nr-of-members = 1`.
 - Failure detection: 2s heartbeat, 10s threshold. Total failover ~25s (drill-measured 2026-07-21 under auto-down: active-crash TAKEOVER in 28s, standby-crash removal in 27s with 0 routing blips — `docker/failover-drill.sh`).
 - CoordinatedShutdown for graceful singleton handover.

docs/plans/2026-07-08-deferred-work-register.md

@@ -61,6 +61,8 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round
 | 65 | **`BundleUnlockAttempt` rows are pruned only by nodes actively serving unlocks** *(Low)* | Row-52 implementation trade | The prune rides the unlock path, so a cluster that stops importing entirely leaves the last rows in place indefinitely (bounded and tiny: one row per permitted attempt, retained 2× the window). | The table appears in a size review, or Transport gains a central scheduler a prune tick could ride |
 | 66 | ~~**No counter for import-session cap rejections** *(Low)*~~ **RESOLVED 2026-08-15 (`90ce0999`).** `scadabridge.central.transport.import_sessions.rejected` on the existing allowlisted meter, emitted ONLY at the post-eviction recheck — the cap is tested twice and a store full of expired sessions reclaims and succeeds, so counting the first test would fire routinely on any node whose sweep hadn't run; duplicate-session-id re-`Open` is also excluded. New tests use delta-over-scoped-action with unique per-test keys (the pre-existing gauge tests' absolute asserts are row 74). | Row-53 implementation note | Closed 2026-08-15. | — |
 | 67 | ~~**Startup reconcile stores a fetched instance config but does not start the InstanceActor** *(Medium-Low, availability)*~~ **RESOLVED 2026-08-15 (`482165cb`).** Root cause: InstanceActors are created from exactly two paths — the boot-time SQLite load in `DeploymentManagerActor.PreStart` and an explicit deploy/enable command — and the reconcile pass fires ~5s AFTER the boot load with a pure file write (`StoreDeployedConfigIfNewerAsync`) as its only side effect. Fix: reconcile collects the enabled instances it stored and Tells a new `EnsureInstanceStartedCommand` to the DM **singleton proxy**, so the actor is created under the singleton no matter which node's pass found the gap. Handler is advisory and idempotent: no-op if already running; re-reads the durable row (authority on config body AND `is_enabled`, so a disable landing after the fetch wins); defers to `_deployWarms`/`_terminatingActorsByName`/`_inFlightDeployments` so deploy-gate ordering and all-or-nothing-per-instance are untouched (it starts an already-persisted config, never applies one); quiet no-op on an absent row (normal on an unreplicated site whose peer healed its own file). The heal is no longer silent — a Warning site event `Instance <X> started by startup reconcile self-heal` fires. Fail-before by behavioral revert: 3 of 8 new tests fail pre-fix, including the exact live symptom (`debug snapshot` → `instanceNotFound`); the 5 negative guards (disabled / failed-fetch / unknown / idempotence) green in both states. SiteRuntime 613/613, Host 504/504. Sibling gaps found during the fix recorded as rows 72/73. | Proven live during the rows-46–53 close-out (2026-08-15) | Closed 2026-08-15. | — |
+| 69 | ~~**Both site-a nodes' LocalDb files corrupt — replication down, one node crash-looping**~~ **RESOLVED 2026-08-15 (operational recovery, no code change).** Found live during this wave: site-a-b threw `SQLite Error 11: 'database disk image is malformed'` on every sync call (replication `connected: false`, backlog 892, faulted session retried ~every 60s) and, after the redeploy, site-a-a crash-looped (SIGABRT ×12) on the same error at `SqliteLocalDb` construction — the new boot-time re-registration reads pages the old build never touched, so site-a-a's corruption (present but latent) became fatal only on upgrade. Recovery: pair stopped together, both files quarantined (`*.corrupt-2026-08-15`, kept for forensics; 394 MB / 416 MB), fresh files reseeded — site-a-b via its startup reconcile (which live-proved the row-67 fix, see row 67) and CDC snapshot resync; converged to `connected: true`, backlog 0, with the row-27 schema change proving the digest handshake green. Suspected cause is host-level, not product: see row 79. | Found live by the row-55 debug + this wave's close-out redeploy | Closed 2026-08-15. | — |
+| 70 | **`auto-down-unreachable-after` is deprecated — Akka warns "keep-majority will be used instead" on every boot** *(Medium info — verified NOT currently true, but a live removal risk)* | Row-55 live session; investigated + drill-verified at this wave's close-out | The warning is real (fires at Cluster start on every node, Akka.NET 1.5.62 — same version as the 2026-07-21 decision, so nothing changed underneath; it fires on the presence of the `auto-down-unreachable-after` key). **Drill-proven that the substitution does NOT happen when `downing-provider-class = "Akka.Cluster.AutoDowning"` is EXPLICIT (as `BuildHocon` sets):** `DRILL_MODE=active` kills central-a (oldest AND lowest address) — under keep-majority a 2-node even split keeps the lowest-address side, so surviving central-b would have downed ITSELF (total outage); instead central-b observed OldestChanged → became Oldest, took over all singletons, and the victim rejoined ready 2s after restart (2026-08-15 drill, post-merge rig). The 2026-07-21 decision therefore still holds exactly as documented. The residual risk is the deprecation itself: a future Akka.NET bump can delete `AutoDowning`, silently or loudly breaking the availability-first strategy. | **Any Akka.NET version bump** — re-run `DRILL_MODE=active bash docker/failover-drill.sh` and re-read the release notes before accepting it |
 | 71 | **Standby nodes log the active-node health check at ERR every 10s** *(Low, log noise)* | Row-55 live session (2026-08-15): site-a-b logs `Health check active-node with status Unhealthy … 'not the site pair primary (standby)'` at Error level every 10s | Correct-by-design standby behavior emitted at an alarming severity — floods the standby's log and trains operators to ignore ERR. Candidate: filter/downgrade the ASP.NET health-check publisher's severity for the expected-standby case. | Next observability/log-hygiene session |
 | 72 | **Reconcile refreshes a STALE config but does not restart the running InstanceActor** *(Medium-Low — same silent-divergence class as row 67, arguably wider blast radius)* | Row-67 fix session (2026-08-15): `ReconcileGapItem` covers both *missing* and *stale* (`StaleInstance_IsRefreshed` pins a revision bump); for a stale item the new config is written to SQLite while the running actor keeps serving the OLD compiled config until the next restart or redeploy | Deliberately out of row-67's scope: restarting a LIVE instance from reconcile carries redeploy semantics (stop/start, subscription re-establish, in-flight runs) that need their own design decision. Nothing surfaces "stored revision ≠ running revision". | Next SiteRuntime/reconcile session, or first field report of an instance serving a stale config after a central redeploy raced a site outage |
 | 73 | **`SiteReconciliationActor` is silently skipped when `IDeploymentConfigFetcher` is null** *(Low)* | Row-67 fix session: `AkkaHostedService` (~line 1046) logs only a Warning and spawns no reconciler | The entire self-heal path — file heal AND the new row-67 runtime heal — can be absent with no health-report surface; a misconfigured site looks healthy while unable to heal. Candidate: surface reconciler presence on the site health report. | Next health-report or reconcile session |
@@ -68,6 +70,9 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round
 | 75 | **SecuredWrites Approve renders as a red destructive button** *(Low, UI)* | Row-28 audit (2026-08-15): the call site passes `danger: true`, so Approve is styled destructive; the label is now correct ("Approve") but the styling still signals delete | Approving a secured write DOES execute a device write, so a caution affordance is arguable — which is why row 28 fixed only the label. A deliberate styling decision belongs to whoever owns the page. | Next Secured Writes / Central UI session |
 | 76 | **CLI README's `audit-config query` JSON sample omits `afterStateJson`/`bundleImportId`** *(Low, docs)* | Row-57 fix (2026-08-15): the real payload returns both fields; the sample doesn't show them | Misleading as written; one-line doc fix, out of row-57's table-projection scope. | Next CLI docs touch |
 | 77 | **Test pattern: reflecting `ActorCell.Actor` right after `ResolveOne` is unordered vs child construction** *(Low, test-infra)* | Row-40 session (2026-08-15): `InstanceActorChildAttributeRaceTests` failed ~1-in-6 under full-suite load once the sleeps stopped padding the schedule (latent — A/B'd 14/14 green on the unmodified tree); fixed with a bounded-deadline resolution retry | `ResolveOne` throws until the child's cell publishes on the parent, and `cell.Actor` stays null until the child's ctor runs on its own mailbox thread — neither ordered against the test thread. The one instance is fixed; the pattern may exist elsewhere. | Next time a child-resolution test flakes under load |
+| 78 | **Configs arriving via LocalDb replication start no InstanceActor — row 67's gap through a side door** *(Medium-Low, availability; proven live 2026-08-15)* | This wave's close-out: site-a-b's reconcile fetched 4 configs at its solo boot; when the pair was then restarted with a fresh site-a-a file, CDC snapshot replication seeded site-a-a's `deployed_configurations` within ~5s of boot — BEFORE its own reconcile ran, so central rightly reported no gap ("0 fetched"), and nothing started the actors: enabled instances sat dead with their configs present (verified `instanceNotFound`) until one more pair restart | The row-67 fix covers configs the reconcile itself fetches; configs that arrive by replication (fresh standby seeded from a peer, or any post-boot snapshot resync) hit the boot load too late and the reconcile not at all. Fix shape: the row-67 `EnsureInstanceStartedCommand` machinery already exists — the missing piece is a trigger when replication writes a `deployed_configurations` row for an enabled instance with no running actor (e.g. a periodic sweep in the DM singleton, or a LocalDb change hook). | Next SiteRuntime/reconcile session — pairs naturally with row 72 |
+| 79 | **OrbStack virtiofs is a live corruption + availability hazard for the rig's bind-mounted SQLite files** *(Medium, rig infra — host-level, not product)* | This wave's close-out: while verifying, the OrbStack VM idle-SLEPT mid-session (`vmgr … msg=sleep`) leaving docker hung/empty for ~20 min until a manual `orb restart docker`; on wake the virtiofs layer logged repeated `refresh_nodeid: race with new lookup` ERRORs. Both site-a LocalDb files (row 69) were corrupted through this same bind-mount layer — macOS virtiofs does not give SQLite true fsync durability, and abrupt container kills (`--force-recreate`) mid-write compound it | Every ScadaBridge data file on the rig (`site-localdb.db`, `auditlog.db`, secrets) rides a relative bind mount through virtiofs. Candidates: move rig data dirs to named docker volumes (native ext4 inside the VM — but loses host-side inspectability the sessions rely on), disable OrbStack idle-pause, or accept + document the quarantine-and-reseed recovery (now proven: row 69). | Next corruption occurrence, or next rig-infra session |
+| 80 | **Unreplicated sites report NON-NULL `LocalDbReplicationConnected: false` / `LocalDbOplogBacklog: 0` — contradicting the documented nullable semantics** *(Low-Medium, observability correctness)* | Found at this wave's close-out verifying row 33: site-b/site-c (deliberately unreplicated) health reports carry `false`/`0`, not null — yet CLAUDE.md and the health docs state "both nullable — null means no data, NOT disconnected with an empty backlog" | Two consequences: a dashboard reader sees `connected: false` on a healthy unreplicated site and reads it as a fault; and row 33's null-branch (conditional KPI emission, badge anchor exclusion) never gets its null on the rig, so unreplicated sites emit misleading `localDbReplicationConnected=0` KPI series (the badge stays safe only because a 0 backlog can't sustain). Root cause is site-side report assembly supplying defaults instead of nulls when replication is unconfigured. Fix: thread `SiteLocalDbSetup.ReplicationIsConfigured` into the health report so unconfigured ⇒ null, and the row-33 machinery then behaves as designed. | Next health-monitoring session |
 | 68 | ~~**`deploy.sh` run from a git worktree silently rehomes every node's bind-mount data dir**~~ **RESOLVED 2026-08-15 (guard added in the same close-out).** The compose files' bind mounts are relative (`./site-a-node-a/data`), so a deploy from a worktree checkout mounts the worktree's copies — the rig keeps running and looks healthy, but its SQLite stores split from the main checkout's and everything written there is orphaned by the next deploy from main. Bit the rows-46–53 close-out live: the seeding agent's `rig-probe-1` deploy persisted into worktree-homed dirs and "vanished" after the orchestrator's redeploy (no tombstone, no product bug — LWW/tombstone forensics ruled out deletion; the row sat in `.claude/worktrees/…/docker/site-a-node-a/data/site-localdb.db` the whole time). Both `docker/deploy.sh` and `docker-env2/deploy.sh` now refuse to run from a `.claude/worktrees/` path (override: `SCADABRIDGE_ALLOW_WORKTREE_DEPLOY=1`). | Found during the rows-46–53 close-out | Closed 2026-08-15. | — |
 
 ## Resolved (verified against the code 2026-07-10)

docs/plans/2026-07-21-auto-down-availability-decision.md

@@ -111,3 +111,21 @@ mis-explanation in the repo ("the alone-oldest is dead and cannot down itself").
 - Monitoring already surfaces dual-active if it ever happens: both nodes report
   `IsActive` in heartbeats / both `/health/active` = 200 — the Health dashboard shows
   two Primaries.
+
+## Postscript 2026-08-15 — the deprecation warning is benign (drill-verified), but pins us for Akka bumps
+
+Every node logs at Cluster start: *"The `auto-down-unreachable-after` feature has been
+deprecated as of Akka.NET v1.5.2 … The `keep-majority` split brain resolver will be used
+instead."* Investigated and drill-verified on 1.5.62 (the same version this decision was
+made and drilled on): the warning fires purely on the **presence of the key** and the
+substitution does **NOT** happen when `downing-provider-class = "Akka.Cluster.AutoDowning"`
+is set explicitly, as `BuildHocon` does. Proof by discriminating drill (`DRILL_MODE=active`,
+2026-08-15): killing central-a — oldest AND lowest-address — left central-b to take over
+(OldestChanged → all singletons moved, victim rejoined ready 2s after restart). Under
+keep-majority, a 2-node even split keeps the lowest-address side, so central-b would have
+downed ITSELF; it demonstrably did not. The decision stands exactly as written.
+
+The real exposure is forward-looking: `AutoDowning` is deprecated and can be REMOVED by a
+future Akka.NET release. **Any Akka version bump must re-run `DRILL_MODE=active
+bash docker/failover-drill.sh` and re-read the release notes before being accepted**
+(deferred-work register row 70).