fix(deploy+cli): review findings — honest CLI timeouts, watermark-complete staleness, phase-2 staging, lock-safe cancellation

Six adversarial-review findings, each verified against the code first.

F1 (HIGH) CLI HttpClient capped every call at min(30s, caller timeout),
silently truncating deploy site's 5-minute BulkDeployTimeout and the
5-minute bundle export/preview/import calls — which printed a fake
"504 Request timed out" while the server kept working. HttpClient.Timeout
is now Timeout.InfiniteTimeSpan (the per-call CTS is the single overall
deadline, connect included) with the connect phase bounded separately on
SocketsHttpHandler.ConnectTimeout. The env override is renamed to
SCADABRIDGE_HTTP_CONNECT_TIMEOUT_SECONDS to match its new meaning.

F2 (HIGH) StaleInstanceProbe's process-static memo served stale hashes
because nothing bumped the watermark on three paths:
  (a) BundleImporter commits through the raw DbContext, so no import ever
      moved the watermark — a second import overwriting the same template
      could be OMITTED from ImportResult.StaleInstanceIds. It now bumps
      once per apply ATTEMPT: after the commit, and after the rollback too
      (the probe runs pre-commit, so a rolled-back attempt leaves memos for
      state that never landed; bumping on both paths is the simplest
      correct shape, versus threading transaction awareness through a
      process-static cache).
  (b) CollectWatermarkBumps' default: arm silently no-op'd, contradicting
      its own doc. It now sets unattributed=true — an over-broad bump costs
      extra work, a missed one produces stale work.
  (c) DataConnection edits route through SiteRepository.SaveChangesAsync,
      which had no watermark at all, yet Protocol/Primary+Backup config/
      FailoverRetryCount are revision-hash inputs. It now bumps (BumpAll —
      a connection has no owning template) after a commit that touched one.

F3 (MED) CLI TemplateTableProjection read child ARRAYS, but ListTemplates
now returns database-projected TemplateSummary rows, so template list
printed all zeros. It now prefers the *Count scalars and falls back to
array length (template get still returns full entities). --detail help
text and README corrected: a listing cannot yield definitions, so --detail
renders the raw summary payload and template get --id is the full dump.

F4 (MED) DeploySiteAsync staged every PendingDeployment in phase 1 against
a 5-min TTL while phase 2 reached them one batch at a time, so tail
instances' fetch tokens could expire before their command was sent.
Staging moved into phase 2, immediately before each send; prepare keeps
its flatten/validate/record work. The staging write is the phase's only
repository touch and is serialised behind a 1-permit semaphore, so the
non-thread-safe DbContext constraint holds and the sends stay concurrent.

F5 (MED, latent) DeploySiteAsync leaked every held operation lock if
cancelled — a wedged per-instance semaphore is permanent for the process.
Phase 2 no longer throws (cancellation is recorded as a per-instance
outcome so phase 3 still runs), and an escape from phase 1 or 3 now
unwinds every unfinalised entry: Failed status + lock release.

F6 (LOW) ScriptCompileVerdictCache's promotion wrote hot directly,
bypassing SegmentCapacity (true ceiling 3x against a documented 2x).
Promotion now goes through Store, keeping generational semantics; _hot
and _cold are volatile.

Tests: CLI 396, DeploymentManager 133, ManagementService 494,
TemplateEngine 478, ScriptAnalysis 60, Transport 157, Transport
integration 106, ConfigurationDatabase 366 — all green, 0 build warnings.
The F2/F4/F5 regression tests were each confirmed to FAIL with their fix
reverted.
This commit is contained in:
Joseph Doherty
2026-08-14 23:51:04 -04:00
parent b1de9dfdd4
commit e0e4b24679
16 changed files with 1172 additions and 174 deletions
+16 -11
View File
@@ -86,28 +86,33 @@ Exit codes:
#### `template list`
List all templates. **Table** output (`--format table`) shows a compact summary — id,
name, description, parent, and member **counts** (`#attrs`, `#alarms`, `#scripts`,
`#comps`, `#nativeAlarms`) — so it stays readable in a terminal. Add `--detail` to dump
the full attribute/alarm/script/composition definitions in the table. **JSON** output
(`--format json`) is always the full, unmodified payload regardless of `--detail`.
List all templates. The server projects this listing in the database and returns
**summary rows** — no child collections, member counts only. **Table** output
(`--format table`) renders them as id, name, description, parent, and member **counts**
(`#attrs`, `#alarms`, `#scripts`, `#comps`, `#nativeAlarms`), so it stays readable in a
terminal. `--detail` skips that column projection and renders the summary payload
verbatim; it does **not** add attribute/alarm/script definitions — a listing never
carries them. Use `template get --id <int>` for a template's full definition. **JSON**
output (`--format json`) is always the unmodified payload regardless of `--detail`.
```sh
scadabridge --url <url> template list # compact table
scadabridge --url <url> --format table template list --detail # full table dump
scadabridge --url <url> --format json template list # full JSON (always)
scadabridge --url <url> template list # compact table
scadabridge --url <url> --format table template list --detail # raw summary rows
scadabridge --url <url> --format json template list # unmodified JSON (always)
```
| Option | Required | Description |
|--------|----------|-------------|
| `--detail` | no | Include full definitions in table output (no effect on JSON) |
| `--detail` | no | Render the raw summary payload instead of the compact column projection (no effect on JSON; does not add definitions) |
| `--skip` | no | Offset paging: number of items to skip (default 0) |
| `--take` | no | Offset paging: max items to return (clamped 1..1000; omit for unlimited) |
#### `template get`
Get a single template by ID. Like `template list`, table output is a compact summary
unless `--detail` is supplied; JSON output is always full.
Get a single template by ID. Unlike `template list`, this returns the **full** template
entity with every attribute, alarm, script and composition inline. Table output is a
compact summary (the same counts as `template list`) unless `--detail` is supplied, which
dumps the full definitions; JSON output is always full.
```sh
scadabridge --url <url> template get --id <int> [--detail]