Commit Graph

69 Commits

Author SHA1 Message Date
Joseph Doherty 3ca968e26b docs(deploy): PROD site-par MxGateway connection re-pointed to the PAR gateways (not installed yet)
Claude-Session: https://claude.ai/code/session_01GT67AD5mc2BnaaAfY3hqqQ
2026-11-20 17:23:23 -05:00
Joseph Doherty 530eaf32c1 docs(deploy): PROD cluster config - Site site-par (id 1), mxgw key scadabridge-prod shared vp03/vp04, MxGateway connection id 1 Connected
Claude-Session: https://claude.ai/code/session_01GT67AD5mc2BnaaAfY3hqqQ
2026-11-20 16:37:41 -05:00
Joseph Doherty facf24b295 docs(deploy): PROD 2e0ce81e installed on vp03/vp04 (stopped, gated on DB indexes) + site-par pair running on par-vp03/04
Claude-Session: https://claude.ai/code/session_01GT67AD5mc2BnaaAfY3hqqQ
2026-11-20 15:39:40 -05:00
Joseph Doherty 9109e4e905 docs(deploy): QA vq03/vq04 on 2e0ce81e (#144/#145) + NAM AD login, one window
Claude-Session: https://claude.ai/code/session_01GT67AD5mc2BnaaAfY3hqqQ
2026-11-19 16:13:55 -05:00
Joseph Doherty 98a8df1f9d deploy(vd03): 2e0ce81e (#144/#145) + NAM AD login on DEV 2026-11-19 15:17:16 -05:00
Joseph Doherty 1213a74353 deploy(#116): DEV vd03 central + site on leader-auto-down (config only)
One key per file: ScadaBridge:Cluster:SplitBrainResolverStrategy auto-down ->
leader-auto-down in appsettings.Central.json and appsettings.Site.json, build
061432c8 unchanged. G6 boot lines on both services, health 200, MxGateway and
Z28062 at baseline, bogus key 401, Overview 6/6 Up. Downtime ~23 s central,
~30 s site. No drill: single-node clusters.

Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
2026-11-15 19:46:47 -05:00
Joseph Doherty 50a5291b64 deploy(#143): DEV vd03 + QA vq03/vq04 to 061432c8 — hash chain removed (migration 65), #116 QA site-pair drills
DEV: ~48 s site / ~29 s central outage; QA: pair stop/start, ~2m40s site / ~1m37s central.
Chain schema + chainer role gone, env key removed, all gates pass on both tiers.
#116 QA site drills (all reactors idle): kill → takeover +7.7 s; stop-and-hold →
[leader-auto-down] DOWNING +27.2 s, takeover +27.4 s, recovery restored byte-identical.

Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
2026-11-15 19:36:04 -05:00
Joseph Doherty 061432c8c4 deploy(#52/#142): Z28061 redeployed on QA (idle, verified); DEV skipped — idle attributes Bad quality
Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
2026-11-15 18:08:30 -05:00
Joseph Doherty 997e36a2bc deploy(#116): QA stop-and-hold drill proves the leader-auto-down timer path
vq04 (active central) killed with SCM auto-restart disabled and held down 94 s:
UNREACHABLE +11.4 s, [leader-auto-down] DOWNING +26.4 s, survivor vq03 active
+26.7 s. Recovery config restored byte-identical before Start-Service; vq04 rejoined
as standby in ~4.7 s with no exit-75. Also corrects the earlier Drill A 'sampler gap'
(an awk keying error; vq04 was ready at +20.8 s).

Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
2026-11-15 18:02:21 -05:00
Joseph Doherty cae5ab958c deploy(#116): QA central kill drill (G14 central) — takeover ~10 s via new-incarnation path
Both centrals killed in turn (Stop-Process on the service PID). SCM restart in ~5 s;
the survivor downed the old incarnation at ~+9-10 s and took the active role; the
victim rejoined as standby at ~+21 s. leader-auto-down stood aside (already Down),
so its 15 s timer path was not exercised on QA. Site-pair drill still pending
(Z28061 slot, #142).

Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
2026-11-15 17:53:09 -05:00
Joseph Doherty 125db03232 deploy(#52): DelmiaReceiver child-name fix applied on DEV + QA (Z28061 excluded)
CvdReactor.ProcessRecipeDownload now addresses Children["DelmiaReceiver"] on DEV vd03
and QA vq03/vq04. Redeployed DEV Z28062 + Z28061Sim, QA Z28062-Z28069; Z28061 left
stale on both tiers (shared with another team, no slot agreed). Traced read-only call
on Z28062 (ReadyFlag False): the pre-fix 'DelmiaRecv.ReadyFlag not found' warning is
gone after the fix.

apply.sh: every CLI call passes --url/--username/--password explicitly (an env-only
SCADABRIDGE_URL silently reached DEV), plus SB_EXPECT (refuse unless the expected
instances exist = tier proof) and SB_SKIP (never deploy the listed instances).

Record: docs/deployment/deployment-records/2026-11-15-gate-52-delmiareceiver-fix.md

Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
2026-11-15 17:25:37 -05:00
Joseph Doherty b19836788b docs(deploy): record QA vq03/vq04 audit hash chain enablement (#107, #139)
SourceNode backfill 843 rows (0 NULL), AuditLog__HashChain__Enabled=true on both
centrals (ChainBacklog unset, head export off, single login), pair stopped together and
started vq03 first: central API ~29 s, sites not restarted. One anchor at
2026-11-13T14:07:43.3208169Z, 0 pre-cutoff rows chained, 20 new rows ChainSeq 1-20,
verify-chain 2026-11 Intact on both centrals, 2026-10 NotChained. MxGateway 2196 Good,
WarmCanary 2 Good, LocalDb sync connected. Backup ScadaBridge_pre-hashchain_20261113T090524.bak.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 09:13:00 -05:00
Joseph Doherty 933da568e3 docs(deploy): record QA WarmCanary OPC UA connection (vq03/vq04 OtOpcUa) and the #138 live gate
Connection 2 / template 27 / instance 16 on site-a, bound to the OtOpcUa script-runner
canaries; serviceLevelFailover at its defaults. Under Warm the connection moved off the
Standby by ServiceLevel (connect-time preference, 0.1 s) and on the booting Standby (50);
a graceful Primary stop still took the connectivity path (66 s, #141). Kept as monitor.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 09:01:19 -05:00
Joseph Doherty 8ba5d5d7fd docs(deploy): record QA vq03/vq04 0e540291 -> 8e32f353, migrations 58->64, #116 leader-auto-down (#107, #116)
Pair window per pack 8ffa6394: COPY_ONLY backup + VERIFYONLY, stop both pairs,
idempotent EF script 58->64 from the Mac (flip in ~2 s), swap, centrals then
sites started together. Site data path ~2 min 6 s, central API ~58 s.
leader-auto-down started on all 4 nodes; no #116 drills; hash chain off;
recovery flag TRUE on all 4 (actions kept). Env digest-identical; only the
strategy key changed in appsettings; per-node JWT/idle values preserved.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 08:14:03 -05:00
Joseph Doherty 581e25caaf docs(deploy): record DEV vd03 424db155 -> 8e32f353, #131 applied, audit hash chain enabled (#107, #139)
No migration (64). ~45 s site outage. SCM non-crash recovery flag set for exit 75 (#132).
#131: SBTEST01/02 AlarmStatus on ScriptAlarm.Priority, verify 13/13 before and after.
Hash chain: SourceNode backfill 843 rows, Enabled=true (ChainBacklog unset, head export off),
one anchor at 2026-11-13T12:16:53Z, 0 pre-cutoff rows chained, verify-chain 2026-11 Intact.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 07:21:31 -05:00
Joseph Doherty 8ffa639477 docs(deploy): refresh QA vq03/vq04 pack to main 8e32f353 - no new migration, owner's go scope (#107, #116)
- Target 3a59c3f6 -> 8e32f353 (adds #138 ServiceLevel failover, #139 chain
  cutoff, #140 graceful stop, LocalDb 0.3.8). No migration in the range;
  the idempotent 58->64 script regenerated at 8e32f353 is byte-identical
  (SHA-256 488da93d...c34ecc0).
- Owner's go 2026-11-13 ("just deploy"): #116 leader-auto-down flip and the
  exit-75 service recovery setting in; no #116 drills; hash chain and head
  export off; #52 stays excluded.
- #131 re-confirmed DEV-only against the QA config DB.
- Preflight re-run 07:05 EST: 58 migrations, 3,441 AuditLog rows, EventId
  key, log 63 MB used of 200.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 07:11:46 -05:00
Joseph Doherty 3badd1f9fe docs(deploy): QA pack hash-chain enablement uses the #139 cutoff (52da02a7, no migration)
Section 7: the enablement needs a build containing 52da02a7 (3a59c3f6 would
still chain the backlog); keep ChainBacklog unset; verify the anchor and that
no pre-cutoff row is chained; enablement-only rollback.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 03:15:49 -05:00
Joseph Doherty 16209831ca docs(deploy): QA vq03/vq04 upgrade pack 0e540291 -> 3a59c3f6, migrations 58 -> 64, #116 bundled (not executed) (#107)
Coordinated central-pair window for #107 item 4: range analysis, ordered
idempotent migration script (SHA-256 488da93d...), read-only SQL preflight
results from wonder-sql-vq03, procedure, gates, rollback, downtime estimate.
QA is on 0e540291 / 58 migrations (not b3021e7c / 57). Hash-chain enablement
after the window is gated on #139.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 02:50:26 -05:00
Joseph Doherty a92a418d7f docs(deploy): prepare the #52 / #131 site-script fixes for DEV and QA (not applied)
Read-only export of DEV vd03 and QA vq03 (2026-11-12): the only
Children["DelmiaRecv"] user is CvdReactor.ProcessRecipeDownload (script 4,
identical on both envs; DEV 3 / QA 9 instances), and the only ScriptAlarm
.Severity reader is DEV TestMesMachine.AlarmStatus (script 42, SBTEST01/02).
QA has no #131 work.

Adds the original (rollback) and fixed bodies, a guarded check/apply/rollback
script that re-sends every script field, and the apply plan: #52 any time,
#131 with or after the dd4f1c2a binary. The old #131 body still compiles on
main but returns banded values; the fixed body is what fails to compile on the
deployed binaries. Rig evidence for both gates is in the README.

Refs #52 #131

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 23:19:07 -05:00
Joseph Doherty 12fa7affc3 chore(deps): LocalDb 0.3.4 -> 0.3.7 (#130, scadaproj#53)
Pins ZB.MOM.WW.LocalDb, .Replication and .Contracts at 0.3.7 (published
2026-11-12). Wire-compatible; no config or code change needed.

- 0.3.6 (scadaproj#36): the library sync service enforces
  LocalDb:Replication:ApiKey itself. LocalDbSyncAuthInterceptor is kept for
  its refuse-every-stream-when-blank rule; its doc comment is updated. The
  rig's site-a pair already carries one identical key on both nodes.
- 0.3.7 (scadaproj#16): a tombstone is held until the peer has it, so a
  node offline past TombstoneRetention no longer resurrects deleted rows.
  The survivor's ledger (sf_messages / site_events deletes) grows for the
  length of the outage; optional TombstoneHardCap bounds it.

Deploy notes added to the topology + installation guides, the S&F
requirement and CLAUDE.md: stop/start the pair together, same ApiKey on
both nodes, ledger growth while a peer is down.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 13:44:53 -05:00
Joseph Doherty 48517b49fb docs: relabel wonder-app-vd03 as DEV, not production, across plans, records, reviews and API specs
vd03 is the DEV box (vd = DEV, vq03/vq04 = QA). Historical docs called it prod/production.
Only vd03-referential uses are changed; generic 'production' wording (production HOCON,
production code paths, ASPNETCORE_ENVIRONMENT=Production, real-world production guidance)
is unchanged. One verbatim git commit message quoted in PLAN-07 is left as recorded.
2026-11-10 15:11:45 -05:00
Joseph Doherty 3a31dd2de7 fix(host): follow an in-place site PSK rotation without a central restart (ScadaBridge#96)
SitePskProvider cached each site's gRPC preshared key forever; only site removal
invalidated it, so rotating SB-GRPC-PSK-{site} broke the central<->site link on
both central nodes until they restarted. The cached key is now dropped when:

- the secret is written through /admin/secrets: central replaces the secrets
  library's ISecretCacheInvalidator with SitePskRotationInvalidator, which also
  publishes SitePskInvalidated on a new DistributedPubSub topic
  (site-psk-invalidation) so the peer central node drops its copy and its
  resolver cache entry too (SitePskInvalidationSubscriber);
- a site rejects a central->site call with PermissionDenied/Unauthenticated:
  WithSiteCredentials wraps every per-site channel's handler in
  PskRejectionRefreshHandler, so the next call re-reads the key (covers
  out-of-process writes such as the secret CLI);
- a site->central call presents a token that does not match the cached key:
  CentralControlAuthInterceptor invalidates and re-reads once before refusing;
- ScadaBridge:Communication:SitePsks reloads.

Invalidation is case-insensitive because SecretName lowercases names. Rotation
runbook (topology-guide), Component-Communication/Host and the #85 cache audit
(D2) updated.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 14:30:17 -05:00
Joseph Doherty 0e89966da6 docs: point markdown backlogs at Gitea issues (status lives in Gitea) 2026-11-06 15:31:44 -05:00
Joseph Doherty 10c04a23d7 docs(deploy): record vd03 + QA rolling upgrade 3808b49f -> 0e540291 (no migration; QA one box at a time, no full outage) 2026-10-22 18:06:26 -04:00
Joseph Doherty 3e1435dfac docs: QA central pair relays standby inbound API calls (StandbyForwardPeerEndpoints) 2026-10-22 16:59:07 -04:00
Joseph Doherty 4bc124d73e docs: QA central pair moved to shared SQL key + secret stores; record the Windows import-rename defect 2026-10-22 15:56:25 -04:00
Joseph Doherty 5ad6f83176 docs(deploy): record QA wonder-app-vq03/vq04 upgrade b3021e7c -> 3808b49f (migration 58, pair-wise window 2m35s) 2026-10-22 15:38:27 -04:00
Joseph Doherty 51b111a35f docs(deploy): record DEV wonder-app-vd03 upgrade b3021e7c -> 3808b49f (migration 58, gate 31/31) 2026-10-22 15:21:39 -04:00
Joseph Doherty 2da90bf151 chore(host): remove unused Database:MachineDataDb option, validation, config and docs 2026-10-22 06:08:18 -04:00
Joseph Doherty d641186397 docs(site-secrets): final-review doc fixes — replication prerequisite, upgrade impact, audit shape
I2: topology guide + Component-SiteRuntime + register row 127 state that a site pair using
deployed secrets must run LocalDb replication. I3: row 127 + topology guide state the upgrade
impact (whole artifact deploy fails and reads Out of date until referenced secrets exist
centrally). M6: audit row carries secrets/missing, no removed field; removal-only deploys write
no central row. M2/M3/I1 as-built notes; CLAUDE.md secrets bullet corrected.
2026-09-28 21:08:34 -04:00
Joseph Doherty 0ccf68951d docs: site secrets ride the artifact deploy; gRPC hub retired in ScadaBridge 2026-09-28 20:22:45 -04:00
Joseph Doherty 2aede140e2 deploy(qa): b3021e7c to vq03/vq04 (55->57, artifacts deployed); SimpleAlarmStatusRequest copied from vd03; register row 126 closed 2026-09-28 11:12:05 -04:00
Joseph Doherty 5cc09b08ef deploy(vd03): b3021e7c — script file access live (AllowedRoots E:\ApiInstall\ScadaBridge\recipes); gate 31/31, rig 8/8
Second vd03 window of the day: no migrations, appsettings.Site.json gains ScriptFileAccess:AllowedRoots
(pushed over SMB before the swap), sample recipes in E:\ApiInstall\ScadaBridge\recipes. The Delmia
test receiver now reads + parses the downloaded file; real HTTP R100 -> 12 values loaded, R900 -> rejected
with line numbers. verify-mes-test-machine.sh: D1's cleared check reads through BTDB, so it only runs
with group C (the rig runs VERIFY_GROUPS=AD).
2026-09-28 10:45:54 -04:00
Joseph Doherty 0af0672597 deploy(vd03): fe89363b + migrations 56/57; in-memory MES/Delmia/alarm-API test machines
vd03 (DEV) upgraded 2b9a7c3c -> fe89363b via the deployment-only winrm channel: COPY_ONLY backup,
55->57 delta script (AddAlarmMesReportable, AddSiteArtifactRevision), ~53 s window, snapshots
identical to baseline, artifacts deployed. Register row 126 now covers the QA pair only.

docker/seed-mes-test-machine.sh (idempotent) builds a standalone TestMesMachine base template -
Left/Right TestMESReceiver + TestDelmiaReceiver (prod attribute names, STATIC), computed alarms
with IsMesReportable, CvdReactor-compatible MesMoveIn/MesMoveOut/ProcessRecipeDownload/AlarmStatus
scripts ported to SetAsync+WaitAsync, and Galaxy-side simulators with reject/no-response knobs -
plus instances SBTEST01/02 and the SimpleAlarmStatusRequest (was a stub) / AlarmStatusRequest
inbound methods in the legacy /mes/(simple)alarmstatus shapes, resolving machines through BTDB.
BTDB rows SBTEST01=990001 / SBTEST02=990002 added on vd03.
docker/verify-mes-test-machine.sh drives the real inbound methods via flow-debug: PASS 25/25.
2026-09-28 10:10:17 -04:00
Joseph Doherty 17864d1ae4 docs(deploy): records for 2b9a7c3c on DEV vd03 and the QA pair vq03/vq04
Both driven from the Mac over the winrm CLI (WinRM 5985 + SMB 445, no helper
on the boxes). Migration 53->55; the first Central boot encrypts
DataConnections.PrimaryConfiguration (dp:protocol:v1:), so rollback is the
COPY_ONLY DB restore, not a binary swap. QA: all four nodes stopped together,
one shared-DB migration, 2 min 35 s window, preflight against a restored clone.

Claude-Session: https://claude.ai/code/session_01RgthVLVU5CJ8SUuEwruTkt
2026-09-10 12:25:53 -04:00
Joseph Doherty 551f753580 docs(deployment): planned record for moving QA (vq03/vq04) and prod (vd03) central inbound key stores to SQL Server; plan doc no longer spells out the dev sa password
Claude-Session: https://claude.ai/code/session_013L3XdmVii9i5sY4FChU8pB
2026-09-02 23:33:38 -04:00
Joseph Doherty f47ceb55b1 docs: inbound API key store — shared SQL Server provider on central pairs (spec, runbook + central-pairs procedure, DBA DDL copy, topology, env inventory, CLAUDE.md, register row 108 closed with rig proof, row 109 Secrets backoff follow-up, Delmia spec §8)
Claude-Session: https://claude.ai/code/session_013L3XdmVii9i5sY4FChU8pB
2026-09-02 23:31:21 -04:00
Joseph Doherty f07ba5cff5 docs(deployment): QA vq03/vq04 + prod vd03 upgraded to afb27e77 (template picker) — binary-only, no migrations; all six health endpoints Healthy
Claude-Session: https://claude.ai/code/session_01Rs5tk49eMrNcwmi8FAohHX
2026-08-21 08:34:49 -04:00
Joseph Doherty a3e2df325c docs(deployment): QA vq03/vq04 + prod vd03 upgraded to 691a2f2f (create-time bindings) — binary-only, no migrations; healthy; outgoing-process stop-timeout FTL noted
Claude-Session: https://claude.ai/code/session_01Rs5tk49eMrNcwmi8FAohHX
2026-08-21 06:36:24 -04:00
Joseph Doherty 8beee522a3 docs(deployment): prod vd03 hotfix da0bfe2b (binary-only swap via WinRM, app.bak.20260820T170309) — healthy, boot-time advise retries match the a24fa38e boot
Claude-Session: https://claude.ai/code/session_01Rs5tk49eMrNcwmi8FAohHX
2026-08-20 17:08:27 -04:00
Joseph Doherty 50f7439e33 docs(deployment): QA hotfix da0bfe2b — MxGateway graceful-stream-end disconnect; incident root cause (dashboard session close 15:29:03) 2026-08-20 16:57:45 -04:00
Joseph Doherty f5a74db3de docs(deployment): record prod upgrade of wonder-app-vd03 to a24fa38e (via WinRM from vq04) — 6 EF migrations, AuditLog:SiteWriter path, System32 auditlog.db relocated; Z28061 270 rows / 8 groups 2026-08-20 16:40:00 -04:00
Joseph Doherty 5b9cf772db docs(deployment): record QA upgrade of wonder-app-vq03 + vq04 (all 4 nodes) to a24fa38e — 6 EF migrations, AuditLog:SiteWriter path added, System32 auditlog.db relocated; grouped bindings verified on Z28061 (264 rows / 8 groups) 2026-08-20 16:31:57 -04:00
Joseph Doherty 25c15dd164 ops(docker)+test(host)+docs: close register rows 104 + 105 — offline VACUUM recipe applied to the rig; build-time pin that no replicated table is a cascading-FK parent; record row 106 (LocalDb as_of_seq=0 full-snapshot loop)
Row 104: docker/vacuum-site-localdb.sh stops both nodes of a pair together, sets
auto_vacuum=INCREMENTAL + VACUUM on each node volume with integrity_check before/after,
then starts both (DRY_RUN=1 reports). Run on all three rig pairs: six files 0 -> 2, ok,
~11% smaller; site-a replication reconnected with backlog 0 and no watermark change —
VACUUM is replication-invisible (ledger keys by declared PK, oplog seq is INTEGER PRIMARY
KEY AUTOINCREMENT). Topology guide gains the recipe.

Row 105: SiteLocalDbReplicatedTableFkGuardTests builds the real site schema via OnReady
(unreplicated config so 0.3.3's own guard cannot mask the culprit) and asserts every
SiteLocalDbSetup.ReplicatedTables entry (now internal) exists and has no inbound
CASCADE/SET NULL/SET DEFAULT foreign key, with a negative control. The constraint is
enforced at build instead of at the first replicated boot.

Row 106 (found during 104's live verification, NOT caused by it — the identical 127k-row
full snapshot also landed at the earlier deploy restart): SnapshotStreamer.SendAsync stamps
as_of_seq = MAX(seq), which is 0 once a converged sender's oplog is ack-pruned to empty;
a zero-watermark receiver (rebuilt node) is never lifted, so the empty-oplog horizon rule
re-streams a full snapshot on every reconnect. Sender-side one-line fix for LocalDb 0.3.4:
as_of_seq = max(MAX(seq), last_acked_seq).
2026-08-20 09:37:48 -04:00
Joseph Doherty 324ba369d9 infra(docker)+test(siteeventlog): close register tripwires 79 + 93 — rig data to named volumes; pin site_events schema to the cap-model calibration
Row 79 executed its own prescribed remedy without waiting for a second
corruption: all ten rig data dirs (8 docker/ nodes + 2 docker-env2/ site
nodes) move from virtiofs bind mounts to named docker volumes
(scadabridge-<node>-data), seeded in place with both rigs stopped and
live-verified (site-a localDbReplicationConnected=true / backlog 0 over
the migrated files; env2 stable, 0 restarts). The feared inspectability
loss is moot — OrbStack exposes volumes at ~/OrbStack/docker/volumes/.
The rebuild surfaced latent env2 drift (pre-Secrets-0.6.2 image): env2
now carries the x-secrets-store-env posture + central data volumes.

Row 93's silent maintenance trap is now loud: SiteEventsSchemaCalibrationTests
pins the site_events columns/indexes the EventLogPurgeService size model is
calibrated against, failing with a pointer to the memo §4 recalibration on
any schema drift. Rows 92 (accepted by design) and 94 (standing tripwire)
annotated accordingly.
2026-08-18 08:43:52 -04:00
Joseph Doherty ddb3c32e06 fix(siteeventlog): scope the storage cap to site_events' own pages (register rows 81, 84)
Row 81 — the cap could not be enforced on a replicated node. MaxStorageMb was
compared against page_count-freelist_count over the WHOLE consolidated LocalDb
file, which since Phase 1/2 holds ten tables plus __localdb_oplog /
__localdb_row_version / __localdb_snapshot_state. Every 1000-row trim batch freed
~390 KB of site_events pages and added ~1000 oplog rows + ~1000 tombstones to the
same file, so the measured size did not fall, the "size did not decrease" guard
fired, and the trim stopped 2.5x over a 64 MB cap with 226 k rows still present.
A node whose peer was down could not enforce its cap at all, and the event log's
budget was in practice shared with bookkeeping it does not own.

The cap now measures site_events AND ITS FOUR INDEXES only, so replication
bookkeeping can neither consume nor block the budget — and the cap stops being a
function of the peer's health, which was the wrong direction of coupling: an
inflated oplog is a replication fault, and letting it drive event deletion would
destroy operational history in response to one.

dbstat — the exact per-table measurement the register proposed — is NOT usable
here, verified before implementing. It is a compile-time option and the bundled
SQLitePCLRaw.lib.e_sqlite3 (SQLite 3.53.3) does not enable it: PRAGMA
compile_options lists no ENABLE_DBSTAT_VTAB, pragma_module_list returns ten
modules with neither dbstat nor sqlite_dbpage, and selecting from dbstat fails
with "no such table". Switching SQLite builds was rejected — the provider is
shared with ZB.MOM.WW.LocalDb and every other consumer, so it trades cap accuracy
for fleet-wide version skew. The footprint is therefore modelled from sampled row
width (500 rows each end, octet_length so multi-byte UTF-8 is not under-counted)
through a page-layout model; whole pages rather than a flat rate, because page
fill collapses once rows approach half a page and a linear model underestimates
that by ~40%. Clamped by whole-file logical size, a true upper bound that can only
correct an over-estimate and never binds on a replicated node.

Index pages count against the cap deliberately: measured at ~34% of the table's
footprint, and the 1 GB default was sized against a 393 B/row figure that already
included them.

Guard semantics restated — two backstops replace the single on-disk check:
(1) row width unsampleable => refuse to trim (checked before the cap comparison,
since a zero width makes "over cap" unanswerable rather than false; the old loop's
answer to an unanswerable cap was to keep deleting); (2) estimated size not
falling across a batch => stop. (2) is the old guard re-based onto the
table-scoped metric, so it can no longer be tripped by replication bookkeeping and
now means exactly one thing: events are arriving as fast as the trim removes them.
For that to be observable the loop re-reads the row count each batch, while
bytes-per-row is sampled once per tick and held fixed — re-sampling per batch
would jitter the estimate as the trim eats the head of the table and could stall
the loop through guard (2), the same self-defeating measurement as row 81 itself.

Row 84 — LogReplicationBacklogNote moved from per-phase to once per tick, driven
by the shared PurgeBudget. A tick where retention AND the cap trim both deleted
used to emit two identical Information notes; the tick total is also the more
useful number beside an oplog spike.

Measured model error: -0.8% / +8.4% / +12.8% across representative workloads,
biased high on purpose (an early cap forfeits budget; a low one fails at its only
job). Independently cross-checks row 34's harness at 397.2 vs 393 B/row.

Fail-before: reverting the estimator to whole-file semantics fails 5 of the new
tests and reproduces the original signature exactly (remaining=2000
estimate=21725184 cap=11534336 — rows destroyed, cap still unmet); restoring
per-phase notes fails the row-84 test with "collection contained 2 matching items".

No config keys added or renamed, so no rig/deploy changes. Decision record:
docs/plans/2026-08-17-site-event-log-cap-semantics-decision.md

Tests: SiteEventLogging 104/104 (was 93), Host 520/520, solution build clean.
2026-08-17 03:58:41 -04:00
Joseph Doherty 9f418a6aea Merge row 34: site_events purge burst measured (2.73M-row worst case = 2.7x MaxOplogRows) and paced — shared 250k/tick budget, 5-min catch-up, ceiling-hit counter 2026-08-16 04:17:16 -04:00
Joseph Doherty d9a35846d9 perf(siteeventlog): pace the site_events purge burst — per-tick delete ceiling + accelerated catch-up (row 34)
Measured the unbounded purge burst and found it can force a full ten-table
snapshot resync, then bounded it. No purge-exemption path was added — the
deletes are still CDC-captured like any other write, which stays ruled out
by design; what changed is how many a single tick performs.

MEASURED (harness: real EventLogPurgeService over a real LocalDb file with
site_events genuinely RegisterReplicated, plus a two-node loopback gRPC
replication pair; docker rig untouched):

  393 bytes/row incl. indexes => a 1 GB event log holds ~2,731,947 rows.
  Expiring all of them in ONE tick:  285 s, 2,731,947 oplog rows,
  284 MB oplog payload, 2.73 M row-version tombstones (7-day retention),
  i.e. 2.7x LocalDb's MaxOplogRows default of 1,000,000. Crossing that makes
  the 60 s maintenance tick prune the oplog and set needs_snapshot — a FULL
  snapshot of all ten replicated tables, bought by a routine retention purge.

  The resident backlog is produced-minus-drained, so a healthy peer blunts it:
  on loopback the same burst peaked at 391,424 and cleared in 6.0 min. That is
  the problem, not the reassurance — the old safety margin was a function of
  peer throughput and vanished with the peer down. A bounded tick is safe at
  any peer speed.

  Drain is not a bottleneck: 7.6k-19.8k tombstones/s (20k / 500k / 2.73M runs).

SHIPPED:
  - SiteEventLogOptions.MaxPurgedRowsPerTick (default 250,000; 0 = unlimited
    escape hatch, negatives rejected). One SHARED budget across the retention
    purge and the cap trim — two independently-capped phases would let a tick
    emit twice the intended burst. 4x margin under MaxOplogRows, which is also
    shared with every other replicated table's ordinary traffic.
  - SiteEventLogOptions.PurgeCatchUpInterval (default 5 min, validated > 0 and
    <= PurgeInterval). Without it pacing would make retention converge in 11
    DAYS at the daily cadence; with it the worst case converges in ~11 ticks
    (~55 min), every burst bounded. 5 min rather than 1 so ticks cannot outrun
    the peer's drain and re-accumulate the backlog.
  - A ceiling is claimed only when work genuinely remains (bounded EXISTS probe),
    so an expired set landing exactly on the budget does not arm a no-op retry.
  - Visibility: rate-limited (15 min) Warning naming the ceiling and the
    accelerated interval, plus an unthrottled counter
    scadabridge.site.event_log.purge_ceiling_hits — its SHAPE distinguishes a
    backlog converging from a site whose event rate outruns one paced tick.
  - The purge loop is a delay loop rather than a PeriodicTimer, so its cadence
    now anchors to the END of the previous purge. Documented.

ROW-33 INTERPLAY VERIFIED: pacing lowers each spike and lengthens total purge
duration, but a full 250k tick drains in ~23 s (~4 min even at 10x a real
link's penalty) against the 15-minute sustained dwell and a 5-minute gap to
the next tick. The UNPACED 2.73M burst had only 2.5x of margin (6.0 min), so
pacing makes the badge assumption robust rather than merely preserving it.

Docs travel together: Component-SiteEventLogging.md, topology-guide.md
("Reading the replication backlog" + the row-33 badge note), CLAUDE.md's
purge-burst sentence, and a dated measurement/decision note under docs/plans/.

Fail-before: the new behavioural tests do not compile against the pre-fix
service (RunPurge was void); with PurgeBudget neutralised to always-unlimited,
6 of the new tests fail and all 19 pre-existing ones pass.

Tests: SiteEventLogging 93/93, Commons 715/715, HealthMonitoring 115/115,
Host 505/505. Solution build clean.
2026-08-16 04:05:21 -04:00
Joseph Doherty 851b528ca3 docs(deployment): row 79 — OrbStack/virtiofs rig hazard + proven recovery runbook
Documents the host-level hazard behind row 69's LocalDb corruption, all
live-verified 2026-08-15. Scope is the macOS docker rigs only — this is a
property of the filesystem the rig's data files sit on, not a ScadaBridge defect
and not something a production Linux deployment is exposed to.

Every rig data file (site-localdb.db, auditlog.db, secrets) rides a relative
bind mount out of the repo tree, which on macOS crosses virtiofs — and virtiofs
does not give SQLite true fsync durability. `--force-recreate`, which deploy.sh
uses, is exactly the abrupt mid-write kill that turns that into a torn file.

Records both observed failure modes, and specifically how differently they
present, which is the part that makes this hard to spot: site-a-b failed loudly
while running (every sync call throwing, connected:false, backlog 892), while
site-a-a was corrupt LATENTLY and only became fatal on the next upgrade, when a
new build's boot-time re-registration read pages the old build never touched
(SIGABRT crash-loop x12). A clean-looking node proves nothing about its file.
Second mode is the VM idle-sleeping mid-operation — `vmgr ... msg=sleep` in
~/.orbstack/log/vmgr.log, then virtiofs `refresh_nodeid: race with new lookup`
errors and a hung docker until `orb restart docker` (data intact, containers
auto-restart). Written so a future session checks vmgr.log BEFORE investigating
ScadaBridge, which is where ~20 minutes went last time.

Recovery runbook is the one that actually worked: stop the pair together,
quarantine site-localdb.db AND its -wal/-shm sidecars with a .corrupt-<date>
suffix (never delete — forensics; and a stale WAL beside a fresh database is its
own corruption), restart together, then VERIFY convergence rather than assuming
it. Notes the precondition that this is a replicated pair's recovery — it leans
on the startup reconcile plus CDC snapshot resync, so an unreplicated node has
no peer to reseed from and needs a backup instead.

Mitigation options tabulated with their real costs (named volumes = native ext4
but loses the host-side inspectability the debugging sessions lean on;
disabling OrbStack idle-pause = a host setting that cannot be version-controlled
and silently does not exist on a new machine; accept + runbook = current), so
the status quo reads as a choice rather than an oversight.

Docs only — no compose change, no host config change, rig untouched.
2026-08-15 21:34:17 -04:00
Joseph Doherty e8a90f18dc merge: external-system retry policy end-to-end (register row 27) — MaxRetries/RetryDelay now travel artifact→gRPC→site schema→S&F enqueue, CLI --max-retries/--retry-delay-seconds, ArtifactDiff compares them 2026-08-15 17:56:21 -04:00