docs(archreview): round-2 re-review (2026-07-12) + 8 fix plans (86 tasks)
Re-ran all 8 domain reviews at HEAD8c888f13against theb910f5ebbaseline: every round-1 finding source-verified (168 fixed, 0 regressions, 0 false claims); 56 new findings (1 Critical / 4 High / 15 Medium / 36 Low), concentrated in post-baseline code (anti-entropy resync, KPI rollup backfill, live alarm stream) and seams the fixes exposed. Headliners: S&F resync predicate inversion can wipe the delivering node's buffer (02-N1 Critical); resync snapshot exceeds the Akka remoting frame size (02-N2); failover drill kills the one node keep-oldest can't survive (01-N1); unbounded rollup backfill per failover (04-R1); live production API key in untracked test.txt (08-NF1). Adds PLAN-R2-01..08 + .tasks.json manifests and the Round-2 board, P0 list, cross-plan mutexes, and wave order in 00-MASTER-TRACKER.
This commit is contained in:
Joseph Doherty
@@ -6,6 +6,57 @@
|
||||
|
||||
Every plan follows the TDD bite-sized-task format and ships a co-located `.tasks.json` manifest. Execute a plan with the `superpowers-extended-cc:executing-plans` skill (pass the plan path); it reads/updates the `.tasks.json`. **When a plan's status changes, update the table below** — this file is the single source of truth for initiative progress.
|
||||
|
||||
---
|
||||
|
||||
# ROUND 2 (2026-07-12)
|
||||
|
||||
**Source:** the round-2 re-review (all 8 `archreview/0X-*.md` reports rewritten 2026-07-12 at HEAD `8c888f13`): every round-1 finding source-verified (168 fixed, 0 regressions, 0 false claims), plus **56 new findings** (1 Critical / 4 High / 15 Medium / 36 Low) concentrated in post-baseline code. Round-2 plans: **8 plans, 86 tasks.**
|
||||
|
||||
## Round-2 Status Board
|
||||
|
||||
| Plan | Domain | Tasks | Done | Status | Findings coverage |
|
||||
|------|--------|------:|-----:|--------|-------------------|
|
||||
| [PLAN-R2-01](PLAN-R2-01-cluster-host-failover.md) | Cluster, Host & Failover | 11 | 0 | ⬜ Not started | N1→T1–T4 (incl. live drill run + envelope measurement, covers R2-08's NF2); N2→T5–T7 (`needs-user`-adjacent: deploy overlay edits, no git add); N3→T8; N4→T9; N5→T10/T11; N6→R2-08 |
|
||||
| [PLAN-R2-02](PLAN-R2-02-communication-store-and-forward.md) | Communication & S&F + live alarm stream | 15 | 0 | ⬜ Not started | **N1 Critical→T1–T4** (shared oldest-Up predicate, failing-first repro); **N2 High→T5–T7** (chunked resync protocol); N3→T8; N4→T9; N5→T6; N6→T10 (MUTEX w/ R2-07); N7→T11/T12; N8→T13/T14; N9→T15 |
|
||||
| [PLAN-R2-03](PLAN-R2-03-site-runtime-dcl.md) | Site Runtime & DCL | 7 | 0 | ⬜ Not started | N1→T1; N2→T2/T3; N3→T4; N4→T5/T6 (full compile-cache adoption); N5/N6→T7 |
|
||||
| [PLAN-R2-04](PLAN-R2-04-data-audit-backbone.md) | Data & Audit Backbone + KPI rollups | 13 | 0 | ⬜ Not started | **R1 High→T2–T4** (sliced backfill + watermark fast-path); R2→T1; R3→T5–T7; R4→T8; R5→T9; R6→T10/T11 (1 EF migration, build-first gotcha noted); R7→T12; final verify T13 |
|
||||
| [PLAN-R2-05](PLAN-R2-05-templates-deployment-transport.md) | Templates, Deployment & Transport | 9 | 0 | ⬜ Not started | N1→T2–T4 (verdict-cache globals-surface keying); N2→T1; N3→T5 (publisher; subscriber = R2-06); N4→T7; N5→T6 (trust-gate extension, hard-error); N6→T8; docs T9 |
|
||||
| [PLAN-R2-06](PLAN-R2-06-edge-integrations.md) | Edge Integrations | 6 | 0 | ⬜ Not started | N1→T1/T2 (**wires the change-bus subscriber**, closes round-1 U2 partial); N2→T3; N3→T5 (typed SITE_UNREACHABLE); N4→T4; N5→T6 |
|
||||
| [PLAN-R2-07](PLAN-R2-07-ui-management-security.md) | UI, Management & Security | 12 | 0 | ⬜ Not started | N1→T1 (hub throttle); N2→T2/T3 (ForwardedHeaders); N3→T4–T6 (secured-write site scope, ManagementActor lane); N4→T7; N5→T8; N6→T10 (MUTEX w/ R2-02); N7→T9; N8→T11; N9→T12 |
|
||||
| [PLAN-R2-08](PLAN-R2-08-conventions-hygiene.md) | Conventions & Hygiene | 13 | 0 | ⬜ Not started | **NF1 High P0→T1–T3** (`needs-user`: key rotation + secret-file deletion — ldap/sql login files hold LIVE secrets too); NF3→T4; NF4→T5–T8 (T8 = possible live `IOperationTrackingStore` DI gap, verify-then-fix); NF5→T10/T11; NF6→T12; NF7→T13; NF8→T9 (Host duplicates are the LIVE bindings — canonicalize SCE sections first) |
|
||||
|
||||
**Round-2 progress: 0 of 86 tasks.**
|
||||
|
||||
## Round-2 P0 (do first, any order)
|
||||
|
||||
1. **R2-08 T1/T2** — rotate the exposed wonder-app-vd03 API key; delete `test.txt` + the three root credential files (ALL contain live secrets incl. a production sysadmin password). `needs-user`.
|
||||
2. **R2-02 T1–T4** — the Critical: unify the active-node predicate; failing-first buffer-wipe repro then fix.
|
||||
3. **R2-02 T5–T7** — the resync frame-size High (chunked protocol).
|
||||
4. **R2-04 T2–T4** — bound the rollup backfill before the next central failover at data volume.
|
||||
|
||||
## Round-2 Cross-Plan Mutexes & Dedupe Rulings
|
||||
|
||||
| Surface / finding | Ruling |
|
||||
|---|---|
|
||||
| `SiteAlarmLiveCacheService.cs` | R2-02 T10 (coalescing) and R2-07 T10 (aggregator deathwatch/sticky IsLive) must NOT run concurrently — serialize, either order. R2-02 T12/T15 also touch adjacent files; keep the whole live-alarm slice single-lane across plans. |
|
||||
| `ManagementActor.cs` | Single-writer lane (round-1 rule stands): R2-07 T4/T6 and R2-02 T13 (`HandleDeleteSite`) serialize against each other. |
|
||||
| `IScriptArtifactChangeBus` | Publisher Add-fix = R2-05 T5; subscriber wiring + Host comment = R2-06 T1/T2. Independent, no cross-plan blockedBy. |
|
||||
| Failover envelope measurement (report-08 NF2) | Owned by R2-01 T4; R2-08 T10 only repoints the register row. |
|
||||
| Deferral-register consolidation (reports 01 N6 / 08 NF5) | Owned by R2-08 T10/T11. |
|
||||
| Options-validation tail | Split by owner: SiteRuntime knobs = R2-03 T4; Transport knob = R2-05 T8; S&F sweep knobs = R2-02 T9; AuditLog/Host = R2-08 T5–T8. |
|
||||
| SBR keep-oldest active-crash topology gap | Still a registered deferred USER decision — no round-2 plan fixes it; R2-01 documents/measures it, R2-08 T10 moves it into the canonical register. |
|
||||
|
||||
## Round-2 Recommended Waves
|
||||
|
||||
- **Wave 0 (P0):** R2-08 T1–T3 (`needs-user`) ∥ R2-02 T1–T7 ∥ R2-04 T1–T4.
|
||||
- **Wave 1:** rest of R2-02 ∥ R2-03 ∥ R2-05 ∥ R2-06 (all file-disjoint domains); R2-01 T1–T4 (drill + measurement, needs live docker cluster).
|
||||
- **Wave 2:** R2-07 (after R2-02's live-alarm slice lands, honoring the mutex) ∥ rest of R2-04 ∥ rest of R2-01 ∥ rest of R2-08.
|
||||
- Full-suite sweep + docker redeploy at the end, per round-1 practice (watch for stale enum-count lock-in tests — bit PLAN-04 and PLAN-07).
|
||||
|
||||
---
|
||||
|
||||
# ROUND 1 (2026-07-08) — COMPLETE
|
||||
|
||||
## Status Board
|
||||
|
||||
| Plan | Domain | Tasks | Done | Status | Findings coverage |
|
||||
|
||||
@@ -0,0 +1,699 @@
|
||||
# PLAN-R2-01 — Cluster, Host & Failover Round-2 Fixes Implementation Plan
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
|
||||
|
||||
**Goal:** Make the failover story *honest* — rewrite and actually RUN the failover drill so it proves what two-node keep-oldest can deliver (and measures what it cannot), apply the three wonder-app-vd03 overlay edits that were deferred on a factually wrong rationale, and close the round-2 residuals (never-reported metrics staleness, stale leader comments, undrained site singletons).
|
||||
|
||||
**Architecture:** The drill (`docker/failover-drill.sh`) gains a `DRILL_MODE` — `standby` (default, the survivable younger-node crash that PASSes within the ~25s SBR budget) and `active` (the oldest-node crash that keep-oldest cannot survive: the drill *expects* the outage and measures it, documenting the registered deferred user decision instead of codifying a recovery the topology cannot deliver). The in-process envelope measurement reuses `TwoNodeClusterFixture` (production `BuildHocon`, now with production-timing knobs) from the previously-skipped `FailoverTimingTests`. Site-role singleton drains come from generalizing `CentralSingletonRegistrar` into a role-aware `SingletonRegistrar` so `deployment-manager` and `event-log-handler` get the same `PhaseClusterLeave` GracefulStop the seven central singletons got in round 1.
|
||||
|
||||
**Tech Stack:** C#/.NET, Akka.NET 1.5.62 (Akka.Cluster, Akka.Cluster.Tools), xUnit, bash + Docker Compose + Traefik, JSON deploy overlays (PowerShell `install.ps1`).
|
||||
|
||||
Build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Test per-project: `dotnet test tests/<project>`. Docker rig: `bash docker/deploy.sh`.
|
||||
|
||||
**Binding scope rulings (do not deviate):**
|
||||
- The two-node keep-oldest **active/oldest-crash availability gap is a registered deferred USER decision** (master tracker 2026-07-08 "Follow-up discovered during P0 execution"; `SbrFailoverTests.cs:15-23`). No task in this plan changes the SBR strategy or topology — Tasks 1–3 *document and measure* the gap.
|
||||
- The oldest-vs-leader predicate unification inside `SiteCommunicationActor`/`SiteReplicationActor` and everything resync-related is **owned by PLAN-R2-02** — do not touch those files here.
|
||||
- N6 (deferred-work-register consolidation) is **owned by PLAN-R2-08** — Task 7 here corrects only the factually wrong N2 prose in the master tracker.
|
||||
- `/deploy/` is **gitignored** (`.gitignore:48: /deploy/`) — Tasks 5 and 6 edit the on-disk artifact and deliberately do NOT `git add` it (never use `git add -f`); Task 7's tracker correction is the version-controlled record.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Rewrite `failover-drill.sh` — standby-victim default + explicit active-victim gap mode
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** Task 4, Task 5, Task 6, Task 8, Task 9, Task 10
|
||||
**Files:**
|
||||
- Modify: `docker/failover-drill.sh` (full rewrite — current script is 45 lines; lines 12-19 pick the ACTIVE node as victim, which is exactly the crash keep-oldest cannot survive)
|
||||
|
||||
Context (round-2 N1): under the round-1 oldest-member unification, the *active* node IS the oldest. `docker/failover-drill.sh:17-19` kills it, so the shipped drill exercises the one crash two-node keep-oldest is *known* not to recover from (the survivor self-downs; the restarted `central-b` cannot re-bootstrap because both nodes list `scadabridge-central-a` as FIRST seed — `docker/central-node-a/appsettings.Central.json:10-13`, `docker/central-node-b/appsettings.Central.json:10-13` — and only the first seed may self-join to form a new cluster). The drill has also never been run (`docker/README.md:288`). Fix the drill so its PASS criteria match reality per direction.
|
||||
|
||||
1. No unit test (bash script). The verification command is the "test": `bash -n docker/failover-drill.sh` (must exit 0), plus `DRILL_MODE=bogus bash docker/failover-drill.sh` (must exit 2 with the usage error — this exercises the argument guard without needing a cluster).
|
||||
2. Replace the script's entire contents with:
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
# Failover drill against the running docker cluster (bash docker/deploy.sh first).
|
||||
#
|
||||
# ROUND-2 REWRITE (arch-review 01 round 2, N1). The original drill killed the
|
||||
# ACTIVE central node — but under the unified oldest-member semantics the
|
||||
# active node IS the oldest, i.e. the one crash two-node keep-oldest CANNOT
|
||||
# survive (registered deferred user decision, master tracker 2026-07-08;
|
||||
# SbrFailoverTests.cs XML doc). Two modes:
|
||||
#
|
||||
# DRILL_MODE=standby (default) — kills the STANDBY (younger) central node.
|
||||
# The survivable direction: SBR downs the crashed member, the active node
|
||||
# keeps its singletons, and Traefik routing never goes dark. PASS = the
|
||||
# survivor logs the member removal within TIMEOUT_S (budget ~25s+: 10s
|
||||
# failure detection + 15s stable-after) while /health/active stays up.
|
||||
#
|
||||
# DRILL_MODE=active — kills the ACTIVE (oldest) central node. THE EXPECTED
|
||||
# OUTCOME IS A TOTAL CENTRAL OUTAGE: keep-oldest downs the partition
|
||||
# without the oldest, so the younger survivor downs ITSELF (down-if-alone
|
||||
# cannot help — the alone-oldest is dead and cannot down itself), and the
|
||||
# self-downed survivor cannot re-form a cluster alone unless it is the
|
||||
# FIRST seed (both nodes list central-a first; only the first seed may
|
||||
# self-join). This mode measures the dark window and PASSes only when
|
||||
# central recovers AFTER the victim container is restarted. It exists to
|
||||
# make the registered gap observable — not to pretend it is covered.
|
||||
set -euo pipefail
|
||||
|
||||
TRAEFIK_URL="${TRAEFIK_URL:-http://localhost:9000}"
|
||||
TIMEOUT_S="${TIMEOUT_S:-90}"
|
||||
DRILL_MODE="${DRILL_MODE:-standby}"
|
||||
OUTAGE_CONFIRM_S="${OUTAGE_CONFIRM_S:-60}"
|
||||
|
||||
active_container() {
|
||||
if curl -sf -o /dev/null "http://localhost:9001/health/active"; then echo scadabridge-central-a
|
||||
elif curl -sf -o /dev/null "http://localhost:9002/health/active"; then echo scadabridge-central-b
|
||||
else echo "ERROR: no active central node found" >&2; exit 1; fi
|
||||
}
|
||||
peer_of() { [ "$1" = scadabridge-central-a ] && echo scadabridge-central-b || echo scadabridge-central-a; }
|
||||
|
||||
case "$DRILL_MODE" in
|
||||
standby|active) ;;
|
||||
*) echo "ERROR: DRILL_MODE must be 'standby' or 'active' (was '$DRILL_MODE')" >&2; exit 2 ;;
|
||||
esac
|
||||
|
||||
ACTIVE=$(active_container)
|
||||
if [ "$DRILL_MODE" = standby ]; then
|
||||
VICTIM=$(peer_of "$ACTIVE"); SURVIVOR="$ACTIVE"
|
||||
else
|
||||
VICTIM="$ACTIVE"; SURVIVOR=$(peer_of "$ACTIVE")
|
||||
fi
|
||||
|
||||
echo "mode=${DRILL_MODE} active=${ACTIVE} victim=${VICTIM} survivor=${SURVIVOR}"
|
||||
KILL_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ)
|
||||
docker kill "${VICTIM}" > /dev/null
|
||||
START=$(date +%s)
|
||||
|
||||
if [ "$DRILL_MODE" = standby ]; then
|
||||
echo "Standby crash: waiting for ${SURVIVOR} to DOWN+REMOVE the dead member (SBR budget ~25s)..."
|
||||
BLIPS=0
|
||||
while true; do
|
||||
ELAPSED=$(( $(date +%s) - START ))
|
||||
curl -sf -o /dev/null "${TRAEFIK_URL}/health/active" || BLIPS=$((BLIPS + 1))
|
||||
if docker logs --since "${KILL_AT}" "${SURVIVOR}" 2>&1 | grep -Eiq "marking.*node.*down|member removed|is removed"; then
|
||||
echo "PASS: survivor removed the crashed member in ${ELAPSED}s (budget ~25s: 10s detection + 15s stable-after)."
|
||||
echo "Active-node routing blips during the drill: ${BLIPS} (expected 0 — the active node was never touched)."
|
||||
break
|
||||
fi
|
||||
if (( ELAPSED > TIMEOUT_S )); then
|
||||
echo "FAIL: no downing/removal evidence on ${SURVIVOR} after ${ELAPSED}s — SBR did not act" >&2
|
||||
docker start "${VICTIM}" > /dev/null
|
||||
exit 1
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
else
|
||||
echo "Active crash: EXPECTING a central outage (registered keep-oldest gap). Watching /health/active..."
|
||||
DARK_STREAK=0
|
||||
while true; do
|
||||
ELAPSED=$(( $(date +%s) - START ))
|
||||
if curl -sf -o /dev/null "${TRAEFIK_URL}/health/active"; then DARK_STREAK=0; else DARK_STREAK=$((DARK_STREAK + 1)); fi
|
||||
if (( DARK_STREAK >= 10 )); then
|
||||
echo "Outage confirmed at ${ELAPSED}s: no active central node — the younger survivor self-downed"
|
||||
echo "(keep-oldest downs the partition WITHOUT the oldest; this is the registered deferred gap)."
|
||||
break
|
||||
fi
|
||||
if (( ELAPSED > OUTAGE_CONFIRM_S )); then
|
||||
echo "NOTE: /health/active stayed reachable ${ELAPSED}s after killing the oldest — better than the"
|
||||
echo "registered gap predicts. Do NOT celebrate: capture both nodes' logs and investigate before trusting it."
|
||||
break
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
fi
|
||||
|
||||
echo "Restarting ${VICTIM}..."
|
||||
docker start "${VICTIM}" > /dev/null
|
||||
RESTART_AT=$(date +%s)
|
||||
echo "Waiting for central to be routable again through Traefik (${TRAEFIK_URL}/health/active)..."
|
||||
while true; do
|
||||
ELAPSED=$(( $(date +%s) - RESTART_AT ))
|
||||
if curl -sf -o /dev/null "${TRAEFIK_URL}/health/active"; then
|
||||
echo "Recovered: an active central node is routable ${ELAPSED}s after the victim restart."
|
||||
break
|
||||
fi
|
||||
if (( ELAPSED > 120 )); then
|
||||
echo "FAIL: central not routable 120s after restarting ${VICTIM}" >&2
|
||||
exit 1
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
|
||||
echo "Survivor singleton/downing evidence (last 20 matching log lines from ${SURVIVOR}):"
|
||||
docker logs "${SURVIVOR}" 2>&1 | grep -Ei "singleton|oldest|downing|removed" | tail -20 || true
|
||||
echo "Drill complete (${DRILL_MODE}). Verify on the Health dashboard that both nodes show Up and exactly one is Primary."
|
||||
```
|
||||
|
||||
3. `chmod +x docker/failover-drill.sh`. Verify: `bash -n docker/failover-drill.sh` (exit 0) and `DRILL_MODE=bogus bash docker/failover-drill.sh; echo "exit=$?"` (prints the usage error, exit=2).
|
||||
4. Commit:
|
||||
```
|
||||
git add docker/failover-drill.sh
|
||||
git commit -m "fix(docker): failover drill kills the STANDBY by default; active mode measures the registered keep-oldest outage instead of pretending recovery (plan R2-01 T1)"
|
||||
```
|
||||
|
||||
### Task 2: Correct the recovery narrative + document the first-seed bootstrap constraint
|
||||
|
||||
**Classification:** trivial (doc-only)
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** Task 4, Task 5, Task 6, Task 8, Task 9, Task 10
|
||||
**Files:**
|
||||
- Modify: `docker/README.md` (Failover Testing → "Automated failover drill" section, lines 273-288)
|
||||
- Modify: `docs/requirements/Component-ClusterInfrastructure.md` ("Down-if-alone recovery" subsection, lines ~104-116)
|
||||
|
||||
1. `docker/README.md` (lines 273-288): rewrite the drill section to match Task 1's two modes. Specifically:
|
||||
- Replace the four numbered bullets with per-mode descriptions: `DRILL_MODE=standby` (default) = the survivable younger-crash direction — expected result: **no routing outage at all** (the active node is never touched) and member removal on the survivor within **~25s** (10s failure-detection threshold + 15s stable-after; the 2s heartbeat interval is not additive); `DRILL_MODE=active` = the oldest-crash direction — expected result: **total central outage** until the victim container is restarted (the registered deferred keep-oldest decision, master tracker 2026-07-08), then recovery within ~2 min of the restart.
|
||||
- Delete the line-286 promise "Expected failover: **~25s** … plus the Traefik ~5s health-check interval" — it described a singleton *migration* on active-crash that two-node keep-oldest cannot deliver. State per-direction expectations instead.
|
||||
- Replace the line-288 "Observed failover time: _run after next deploy and record here_" placeholder with a two-row results table Task 3 will fill:
|
||||
|
||||
```markdown
|
||||
> **Observed results** (recorded by plan R2-01 T3, cluster @ commit <sha>):
|
||||
>
|
||||
> | Direction (`DRILL_MODE`) | Outcome | Measured |
|
||||
> |--------------------------|---------|----------|
|
||||
> | `standby` (younger-node crash) | _pending T3_ | _pending T3_ |
|
||||
> | `active` (oldest-node crash) | _pending T3_ | _pending T3_ |
|
||||
```
|
||||
|
||||
- Add a short **"Seed-node bootstrap constraint"** note to the same section: only the FIRST seed in `Cluster:SeedNodes` may self-join to form a *new* cluster; both central nodes list `scadabridge-central-a` first, so a lone restarted `central-b` (with `central-a` still down) loops on `InitJoin` forever — never `Up`, never `/health/active` 200. Operator recovery actions: **(1)** restart the dead first-seed node (preferred), or **(2)** restart the survivor with a self-first seed override (env: `ScadaBridge__Cluster__SeedNodes__0=akka.tcp://scadabridge@<self-host>:8081`, `...__1=<peer>`). Explain why the repo does NOT ship self-first ordering per node: with *both* nodes self-first, a simultaneous cold start can let each self-join independently → two one-node clusters that never merge (a cold-start split-brain the identical-seed-order convention exists to prevent). The real remedy is the pending keep-oldest topology/strategy decision (deferred, owner: user).
|
||||
2. `docs/requirements/Component-ClusterInfrastructure.md` (Down-if-alone recovery, ~104-116):
|
||||
- Amend step 4 ("The restarted process rejoins as a **fresh incarnation** (the keep-oldest resolver handles the rejoin cleanly…)"): add the qualifier that this holds **only while a peer holding cluster state is still reachable** — a lone restarted non-first-seed node cannot re-form a cluster (first-seed self-join rule) and waits for its peer.
|
||||
- Add a "Seed-node bootstrap constraint" paragraph after the numbered list (same content as the README note, spec-toned), cross-referencing the registered deferred keep-oldest decision and the operator actions.
|
||||
- Fix the closing line "The docker failover drill exercises this loop end-to-end (kill the active node, assert singleton migration + clean rejoin)" → "The docker failover drill (`docker/failover-drill.sh`) exercises this per direction: `standby` mode proves SBR downing + singleton continuity on the oldest; `active` mode measures the registered total-outage gap and the recovery-on-restart path."
|
||||
3. Verification (doc task — no test): `grep -n "DRILL_MODE" docker/README.md` (≥2 hits), `grep -cn "singleton migration" docs/requirements/Component-ClusterInfrastructure.md` (0 hits in the recovery section), `grep -n "first seed\|first-seed" docker/README.md docs/requirements/Component-ClusterInfrastructure.md` (≥2 hits), `grep -n "Traefik ~5s health-check interval" docker/README.md` (0 hits).
|
||||
4. Commit:
|
||||
```
|
||||
git add docker/README.md docs/requirements/Component-ClusterInfrastructure.md
|
||||
git commit -m "docs(cluster): per-direction failover truth — first-seed bootstrap constraint + operator recovery actions; drop the undeliverable ~25s active-crash promise (plan R2-01 T2)"
|
||||
```
|
||||
|
||||
### Task 3: RUN the drill live (both directions) and record measured timings
|
||||
|
||||
**Classification:** standard (live-cluster verification; ~5 min of work, several minutes of wall-clock waiting)
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** none (exclusive use of the shared docker cluster; writes `docker/README.md`)
|
||||
**Files:**
|
||||
- Modify: `docker/README.md` (fill the Task 2 results table)
|
||||
|
||||
The drill has NEVER been executed (round-2 N1 point 3, `docker/README.md:288`). This task produces the empirical record — the asymmetric result *is* the documentation the deferred keep-oldest decision needs.
|
||||
|
||||
1. Ensure infra + cluster are up and freshly built (drill needs the Task 1 script inside the checkout only — no image change required, but the cluster must be healthy):
|
||||
```
|
||||
cd infra && docker compose up -d && cd ..
|
||||
bash docker/deploy.sh
|
||||
```
|
||||
Wait until `curl -sf http://localhost:9000/health/active` returns 200 (poll; allow ~2 min for first-boot readiness).
|
||||
2. Run direction 1 and capture:
|
||||
```
|
||||
DRILL_MODE=standby bash docker/failover-drill.sh 2>&1 | tee /private/tmp/claude-501/-Users-dohertj2-Desktop-ScadaBridge/d69633c2-b169-4fe1-8fb4-0d8007a16fa1/scratchpad/drill-standby.log
|
||||
```
|
||||
Expected: PASS with member-removal time ≈ 25-30s and `routing blips: 0`. Wait for the victim to fully rejoin (script does this) plus ~30s settle time.
|
||||
3. Run direction 2 and capture:
|
||||
```
|
||||
DRILL_MODE=active bash docker/failover-drill.sh 2>&1 | tee /private/tmp/claude-501/-Users-dohertj2-Desktop-ScadaBridge/d69633c2-b169-4fe1-8fb4-0d8007a16fa1/scratchpad/drill-active.log
|
||||
```
|
||||
Expected: "Outage confirmed at ~Ns" (survivor self-downs; central dark), then "Recovered: … Ms after the victim restart". If instead the NOTE branch fires (`/health/active` stayed up), capture `docker logs scadabridge-central-a` + `-b` and investigate before recording — do not record a result that contradicts the registered gap without evidence.
|
||||
4. Fill the `docker/README.md` results table from the two logs: outcome + measured seconds per direction, and the cluster commit SHA (`git rev-parse --short HEAD`). Keep the raw one-line summary quotes from the script output.
|
||||
5. Sanity-restore: confirm both nodes healthy afterwards — `curl -sf http://localhost:9001/health/ready && curl -sf http://localhost:9002/health/ready` (both 200; the standby's `/health/active` correctly 503).
|
||||
6. Commit:
|
||||
```
|
||||
git add docker/README.md
|
||||
git commit -m "test(docker): record live failover-drill results for both crash directions — standby-crash PASS timing + active-crash measured outage (plan R2-01 T3)"
|
||||
```
|
||||
|
||||
### Task 4: Wire `FailoverTimingTests` to `TwoNodeClusterFixture` — measure the ~25s envelope in-process
|
||||
|
||||
**Classification:** high-risk (real two-node cluster timing test)
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** Task 1, Task 2, Task 5, Task 6, Task 7, Task 8, Task 9, Task 10, Task 11
|
||||
**Files:**
|
||||
- Modify: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/TwoNodeClusterFixture.cs` (StartAsync lines 24-36, StartNode lines 39-65 — add production-timing knobs)
|
||||
- Modify: `tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/ZB.MOM.WW.ScadaBridge.PerformanceTests.csproj` (add a ProjectReference)
|
||||
- Modify: `tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/Failover/FailoverTimingTests.cs` (replace the `Skip` placeholder — currently `[Fact(Skip = "Requires the real two-node failover rig delivered by PLAN-01 …")]` at lines 25-32)
|
||||
|
||||
This task **also covers architecture-review report 08's NF2 by reference** (the skipped perf placeholder): the rig it waits for exists (`TwoNodeClusterFixture`, PLAN-01 T3); wire it up. The fixture hard-codes scaled timings (`HeartbeatInterval` 500ms / `FailureDetectionThreshold` 2s at `TwoNodeClusterFixture.cs:51-52`); to measure the *production* ~25s envelope (2s heartbeat / 10s threshold / 15s stable-after), the knobs become optional parameters with the current values as defaults (additive — `SbrFailoverTests`/`ActiveNodeSemanticsTests` unaffected). Direction note: the measurable crash is the YOUNGER node (the survivable direction); the oldest-crash direction is the registered deferred gap and is *not* a recovery to time — say so in the test's XML doc.
|
||||
|
||||
1. Rewrite `FailoverTimingTests.cs` (this is the failing step — it will not compile until the fixture knobs + project reference exist):
|
||||
|
||||
```csharp
|
||||
using System.Diagnostics;
|
||||
using Akka.Actor;
|
||||
using Akka.Cluster.Tools.Singleton;
|
||||
using Xunit.Abstractions;
|
||||
using ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.PerformanceTests.Failover;
|
||||
|
||||
/// <summary>
|
||||
/// Failover-timing measurement on the real two-node in-process rig
|
||||
/// (TwoNodeClusterFixture, production BuildHocon) at PRODUCTION timings:
|
||||
/// 2s heartbeat / 10s failure-detection threshold / 15s SBR stable-after —
|
||||
/// the CLAUDE.md "total failover ~25s" design envelope.
|
||||
///
|
||||
/// Measures the SURVIVABLE direction only: hard-crash of the YOUNGER node,
|
||||
/// timed to the survivor's member REMOVAL (detection + stable-after + gossip)
|
||||
/// with singleton continuity asserted on the oldest. The oldest/active-node
|
||||
/// crash is NOT a recovery to time — two-node keep-oldest makes the younger
|
||||
/// survivor down itself (total outage; registered deferred user decision,
|
||||
/// see SbrFailoverTests XML doc + master tracker 2026-07-08). Covers overall
|
||||
/// review P2-10 / report-08 NF2 and report-01 round-2 N1's measurement ask.
|
||||
/// </summary>
|
||||
public class FailoverTimingTests(ITestOutputHelper output)
|
||||
{
|
||||
private sealed class EchoActor : ReceiveActor
|
||||
{
|
||||
public EchoActor() => ReceiveAny(msg => Sender.Tell(msg));
|
||||
}
|
||||
|
||||
[Trait("Category", "Performance")]
|
||||
[Fact]
|
||||
public async Task HardKillOfYoungerNode_SbrRemovalWithinDesignEnvelope()
|
||||
{
|
||||
await using var cluster = await TwoNodeClusterFixture.StartAsync(
|
||||
stableAfter: TimeSpan.FromSeconds(15),
|
||||
heartbeatInterval: TimeSpan.FromSeconds(2),
|
||||
failureDetectionThreshold: TimeSpan.FromSeconds(10));
|
||||
|
||||
// Singleton on the oldest (NodeA) — the continuity probe.
|
||||
var manager = cluster.NodeA.ActorOf(ClusterSingletonManager.Props(
|
||||
Props.Create(() => new EchoActor()),
|
||||
PoisonPill.Instance,
|
||||
ClusterSingletonManagerSettings.Create(cluster.NodeA).WithSingletonName("timing-probe")),
|
||||
"timing-probe-singleton");
|
||||
var proxyA = cluster.NodeA.ActorOf(ClusterSingletonProxy.Props(
|
||||
"/user/timing-probe-singleton",
|
||||
ClusterSingletonProxySettings.Create(cluster.NodeA).WithSingletonName("timing-probe")),
|
||||
"timing-probe-proxy");
|
||||
Assert.Equal("ping", await proxyA.Ask<string>("ping", TimeSpan.FromSeconds(30)));
|
||||
|
||||
var victim = Akka.Cluster.Cluster.Get(cluster.NodeB).SelfAddress;
|
||||
var sw = Stopwatch.StartNew();
|
||||
await TwoNodeClusterFixture.CrashNode(cluster.NodeB);
|
||||
await TwoNodeClusterFixture.WaitForMemberRemoved(cluster.NodeA, victim, TimeSpan.FromSeconds(60));
|
||||
sw.Stop();
|
||||
|
||||
output.WriteLine(
|
||||
$"MEASURED: crash -> member removed on survivor: {sw.Elapsed.TotalSeconds:F1}s " +
|
||||
"(design ~25s = 10s detection + 15s stable-after; +gossip margin)");
|
||||
|
||||
// Design envelope: >= stable-after (SBR must not act early), <= 25s design + 15s margin.
|
||||
Assert.InRange(sw.Elapsed, TimeSpan.FromSeconds(15), TimeSpan.FromSeconds(40));
|
||||
|
||||
// Singleton continuity on the surviving oldest node.
|
||||
Assert.Equal("ping-after-crash",
|
||||
await proxyA.Ask<string>("ping-after-crash", TimeSpan.FromSeconds(10)));
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.PerformanceTests --filter "FullyQualifiedName~FailoverTimingTests"` — expect FAIL (compile error: `TwoNodeClusterFixture` unknown; no `heartbeatInterval` parameter).
|
||||
3. Implement:
|
||||
- `TwoNodeClusterFixture.StartAsync` (line 24): signature → `StartAsync(string role = "Central", TimeSpan? stableAfter = null, TimeSpan? heartbeatInterval = null, TimeSpan? failureDetectionThreshold = null)`; pass both new args through the two `f.StartNode(...)` calls.
|
||||
- `TwoNodeClusterFixture.StartNode` (line 39): signature → `StartNode(int port, string role, TimeSpan? stableAfter = null, TimeSpan? heartbeatInterval = null, TimeSpan? failureDetectionThreshold = null)`; lines 51-52 become `HeartbeatInterval = heartbeatInterval ?? TimeSpan.FromMilliseconds(500),` / `FailureDetectionThreshold = failureDetectionThreshold ?? TimeSpan.FromSeconds(2),`. Extend the class XML doc: "Timing knobs default to scaled test values; pass production values (2s/10s/15s) to measure the real failover envelope (FailoverTimingTests)."
|
||||
- `ZB.MOM.WW.ScadaBridge.PerformanceTests.csproj`: add to the existing `<ItemGroup>` of ProjectReferences:
|
||||
|
||||
```xml
|
||||
<!-- Two-node failover rig (TwoNodeClusterFixture) — brings Host + Akka transitively. -->
|
||||
<ProjectReference Include="../../tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/ZB.MOM.WW.ScadaBridge.IntegrationTests.csproj" />
|
||||
```
|
||||
|
||||
4. Run again: `dotnet test tests/ZB.MOM.WW.ScadaBridge.PerformanceTests --filter "FullyQualifiedName~FailoverTimingTests"` — expect PASS in ~45-90s; the test output MUST print the `MEASURED:` line — quote the number in the commit message. Regression: `dotnet test tests/ZB.MOM.WW.ScadaBridge.IntegrationTests --filter "FullyQualifiedName~SbrFailoverTests|FullyQualifiedName~ActiveNodeSemanticsTests|FullyQualifiedName~TwoNodeClusterFixtureTests"` — all green (fixture change is additive).
|
||||
5. Commit:
|
||||
```
|
||||
git add tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/TwoNodeClusterFixture.cs tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/ZB.MOM.WW.ScadaBridge.PerformanceTests.csproj tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/Failover/FailoverTimingTests.cs
|
||||
git commit -m "test(failover): unskip FailoverTimingTests on the two-node rig at production timings — measured Xs vs ~25s design envelope (plan R2-01 T4; covers report-08 NF2)"
|
||||
```
|
||||
|
||||
### Task 5: Apply the wonder-app-vd03 appsettings overlay edits (NodeName + AllowSingleNodeCluster, drop phantom seeds)
|
||||
|
||||
**Classification:** trivial (config-only; NO git commit — see note)
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** Task 1, Task 2, Task 4, Task 6, Task 8, Task 9, Task 10, Task 11
|
||||
**Files:**
|
||||
- Modify: `deploy/wonder-app-vd03/appsettings.Central.json` (`Node` lines 3-7, `Cluster` lines 8-18, `_comment_Cluster` line 19)
|
||||
- Modify: `deploy/wonder-app-vd03/appsettings.Site.json` (`Node` lines 3-9, `Cluster` lines 10-20, `_comment_Cluster` line 21)
|
||||
|
||||
Round-2 N2: the round-1 deferral claimed the artifact "is not tracked in this repo … the artifact directory itself is not present" — **the directory IS present** (merely gitignored, `.gitignore:48: /deploy/`) and has been actively maintained in this working tree (its `_comment_Security` documents the PLAN-07 edit). Apply the two appsettings edits from PLAN-01 Tasks 16/23 directly. Without `NodeName`, every audit row from the only production install stamps a NULL `SourceNode` (defeats `IX_AuditLog_Node_Occurred` + the per-node stuck KPIs); with the phantom seeds, each node dials a nonexistent peer (`localhost:8091` / `localhost:8092`) forever.
|
||||
|
||||
1. `appsettings.Central.json`:
|
||||
- `Node` (after `"Role": "Central",`): add `"NodeName": "central-a",` (matches the docker convention, `docker/central-node-a/appsettings.Central.json`).
|
||||
- `Cluster`: drop the phantom seed `"akka.tcp://scadabridge@localhost:8091"` (keep only the `:8081` self seed) and add `"AllowSingleNodeCluster": true` (flag shipped in PLAN-01 T16: `ClusterOptions.cs:86`, validator `ClusterOptionsValidator.cs:29-35` requires it for a 1-seed list).
|
||||
- `_comment_Cluster` → `"Single-node central install, explicitly acknowledged via AllowSingleNodeCluster (arch-review 01 S12/N2): the previous phantom second seed (localhost:8091) made the node dial a nonexistent peer forever. When a real central node B is added: list BOTH real seeds and remove AllowSingleNodeCluster."`
|
||||
2. `appsettings.Site.json`: same treatment — `Node` gains `"NodeName": "node-a",`; `Cluster.SeedNodes` keeps only `"akka.tcp://scadabridge@localhost:8082"`; add `"AllowSingleNodeCluster": true`; `_comment_Cluster` rewritten equivalently (PRESERVE the existing "Seed ports must never equal GrpcPort (8083)" sentence).
|
||||
3. Verify (this is the "test" — config task):
|
||||
```
|
||||
python3 -m json.tool deploy/wonder-app-vd03/appsettings.Central.json > /dev/null && \
|
||||
python3 -m json.tool deploy/wonder-app-vd03/appsettings.Site.json > /dev/null && \
|
||||
grep -c "NodeName\|AllowSingleNodeCluster" deploy/wonder-app-vd03/appsettings.Central.json deploy/wonder-app-vd03/appsettings.Site.json && \
|
||||
grep -c "8091\|8092" deploy/wonder-app-vd03/appsettings.Central.json deploy/wonder-app-vd03/appsettings.Site.json
|
||||
```
|
||||
Expect: both json.tool exits 0; each file counts 2 for the first grep; each file counts **0** for the phantom-port grep (grep -c exits 1 on zero matches — that is the pass signal here).
|
||||
4. **No git commit** — `/deploy/` is gitignored (`.gitignore:48`); do NOT use `git add -f`. The edit is intentionally working-tree/on-host only; Task 7 commits the version-controlled record, and the change takes effect on the host when the artifact is next synced + services restarted (RUNBOOK step added in Task 6).
|
||||
|
||||
### Task 6: Complete the install.ps1 recovery actions (`sc.exe failureflag`) + RUNBOOK recovery step
|
||||
|
||||
**Classification:** trivial (script/doc-only; NO git commit — see Task 5 note)
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** Task 1, Task 2, Task 4, Task 5, Task 8, Task 9, Task 10, Task 11
|
||||
**Files:**
|
||||
- Modify: `deploy/wonder-app-vd03/install.ps1` (section `# --- 7. Auto-restart on failure`, lines ~171-174)
|
||||
- Modify: `deploy/wonder-app-vd03/RUNBOOK.md` (troubleshooting table, ~line 218 area)
|
||||
|
||||
Finding-vs-disk note (record this honestly): the round-2 report says the `sc.exe failure` recovery actions are missing, but the on-disk `install.ps1` already has them (lines 171-174, all three services) — what is missing is `sc.exe failureflag <svc> 1`. Without the failure-actions flag, the SCM runs recovery actions only after a **crash**; the down-if-alone watchdog's *deliberate* process exit (`IHostApplicationLifetime.StopApplication()` after an SBR self-down — `AkkaHostedService.cs:203-217`) is a non-crash stop and would never be restarted, breaking the recovery contract (`Component-ClusterInfrastructure.md` step 3: "The service supervisor restarts it"). This is the exact second line PLAN-01 Task 20 specified.
|
||||
|
||||
1. In `install.ps1`, extend the existing loop:
|
||||
|
||||
```powershell
|
||||
# --- 7. Auto-restart on failure ------------------------------------------
|
||||
foreach ($svc in 'ScadaBridge-LDAP', 'ScadaBridge-Central', 'ScadaBridge-Site') {
|
||||
sc.exe failure $svc reset= 86400 actions= restart/5000/restart/5000/restart/10000 | Out-Null
|
||||
# Required for the SBR down-if-alone recovery contract
|
||||
# (Component-ClusterInfrastructure.md "Down-if-alone recovery"): without
|
||||
# failureflag=1 the SCM applies the recovery actions only after a CRASH;
|
||||
# the Host watchdog's deliberate process exit after a self-down is a
|
||||
# normal (non-crash) stop and would otherwise never be restarted.
|
||||
sc.exe failureflag $svc 1 | Out-Null
|
||||
}
|
||||
```
|
||||
|
||||
2. `RUNBOOK.md`: add a troubleshooting-table row: `| Service stopped on its own with log line "ActorSystem terminated outside host shutdown" | SBR self-down (down-if-alone). The service recovery actions restart it automatically (requires install.ps1 with sc.exe failureflag — re-run install.ps1 or run the two sc.exe lines from §7 manually on older installs). If it does not come back: Start-Service manually and check network reachability between nodes. |` Also note next to the "auto-restart on failure" mention (line 87) that failureflag is part of the contract.
|
||||
3. Verify: `grep -n "failureflag" deploy/wonder-app-vd03/install.ps1` (3 effective invocations via the loop → 1 hit inside it) and `grep -n "failureflag\|self-down" deploy/wonder-app-vd03/RUNBOOK.md` (≥1 hit). If `pwsh` is installed, syntax-check: `pwsh -NoProfile -Command "[void][System.Management.Automation.PSParser]::Tokenize((Get-Content -Raw deploy/wonder-app-vd03/install.ps1), [ref]$null)"` (exit 0); otherwise the greps suffice.
|
||||
4. **No git commit** (gitignored artifact — same rule as Task 5).
|
||||
|
||||
### Task 7: Correct the factually wrong N2 deferral record in the master tracker
|
||||
|
||||
**Classification:** trivial (doc-only)
|
||||
**Estimated implement time:** ~3 min
|
||||
**Parallelizable with:** Task 1, Task 2, Task 4, Task 8, Task 9, Task 10, Task 11
|
||||
**Files:**
|
||||
- Modify: `archreview/plans/00-MASTER-TRACKER.md` (line 28 trailing sentence; "Deferred during Wave 1 PLAN-01 execution (2026-07-08)" bullet at lines 145-147)
|
||||
|
||||
The deferral rationale — "that production deploy artifact is not tracked in this repo … the artifact directory itself is not present in the repo" — is factually wrong (round-2 N2): `deploy/wonder-app-vd03/` exists in the working tree, is merely gitignored (`.gitignore:48: /deploy/`), and was edited during this initiative (PLAN-07's `_comment_Security`). Depends on Tasks 5-6 having applied the edits.
|
||||
|
||||
1. Rewrite the line-147 bullet: keep the history (what was deferred and why it mattered), then correct the record: the directory **is present but gitignored** (production config kept out of version control — the *tracking* claim was wrong, the *presence* claim was false); the three overlay edits were **applied on-disk 2026-07-12 by PLAN-R2-01 Tasks 5-6** (`NodeName` central-a/node-a, `AllowSingleNodeCluster` + phantom seeds dropped, `sc.exe failureflag` completing the already-present `sc.exe failure` actions). Remaining owner action: sync the artifact to `wonder-app-vd03` and restart services (appsettings take effect on restart; failureflag on re-running install.ps1 §7 or the two `sc.exe` lines manually) — per the RUNBOOK row added by T6.
|
||||
2. Trim the stale trailing sentence of the line-28 Wave-1 summary ("were deferred because that production artifact is not tracked in this repo") → "were deferred on a mistaken not-in-repo rationale; corrected and applied on-disk by PLAN-R2-01 (2026-07-12) — see the Deferred-during-Wave-1 entry."
|
||||
3. Do NOT add rows to `docs/plans/2026-07-08-deferred-work-register.md` — register consolidation (round-2 N6) is owned by **PLAN-R2-08**; this task only stops the tracker from asserting a falsehood.
|
||||
4. Verify: `grep -n "not tracked in this repo" archreview/plans/00-MASTER-TRACKER.md` → 0 hits; `grep -n "PLAN-R2-01" archreview/plans/00-MASTER-TRACKER.md` → ≥2 hits.
|
||||
5. Commit:
|
||||
```
|
||||
git add archreview/plans/00-MASTER-TRACKER.md
|
||||
git commit -m "docs(tracker): correct the wonder-app-vd03 deferral record — artifact is present (gitignored); overlay edits applied on-disk by R2-01 T5/T6 (plan R2-01 T7)"
|
||||
```
|
||||
|
||||
### Task 8: Metrics-staleness for never-reported sites (`FirstSeenAt` anchor)
|
||||
|
||||
**Classification:** high-risk (lock-free CAS state machine)
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** Task 1, Task 2, Task 4, Task 5, Task 6, Task 7, Task 9, Task 10, Task 11
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthState.cs` (add `FirstSeenAt`)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthAggregator.cs` (register branches lines 44-54 and 114-122; staleness guard lines 300-312)
|
||||
- Modify: `docs/requirements/Component-HealthMonitoring.md` ("Metrics staleness" bullet, line ~47)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/CentralHealthAggregatorTests.cs`
|
||||
|
||||
Round-2 N3: `CheckForOfflineSites` guards staleness with `state.LastReportReceivedAt is { } lastReport` (`CentralHealthAggregator.cs:300`), so a site registered via heartbeat only (`MarkHeartbeat` registration leaves `LastReportReceivedAt` null, `:114-122`) is NEVER flagged stale — "stuck from first boot" shows online-with-no-metrics forever. `LastHeartbeatAt` cannot be the anchor (heartbeats keep advancing it); anchor on a new `FirstSeenAt` stamped at registration. No UI change needed: the badge (`Health.razor:209-215`) keys on `IsOnline && IsMetricsStale` regardless of `LatestReport`.
|
||||
|
||||
1. Write the failing tests (append to `CentralHealthAggregatorTests` — reuse the existing `NewAggregator(metricsStaleTimeout:)` and `TestTimeProvider.Advance` helpers, lines 48-59/22):
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Round-2 N3: a site known only via heartbeats (LastReportReceivedAt == null)
|
||||
/// previously skipped the staleness check forever — a report pipeline dead
|
||||
/// from FIRST BOOT showed "online with no metrics" indefinitely. FirstSeenAt
|
||||
/// anchors the window instead (LastHeartbeatAt cannot: heartbeats advance it).
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void HeartbeatOnlySite_NeverReported_IsFlaggedMetricsStale()
|
||||
{
|
||||
var (aggregator, time) = NewAggregator(metricsStaleTimeout: TimeSpan.FromMinutes(2));
|
||||
aggregator.MarkHeartbeat("site-a", time.GetUtcNow()); // registered via heartbeat only
|
||||
time.Advance(TimeSpan.FromMinutes(3));
|
||||
aggregator.MarkHeartbeat("site-a", time.GetUtcNow()); // still heartbeating => online
|
||||
aggregator.CheckForOfflineSites();
|
||||
var state = aggregator.GetSiteState("site-a")!;
|
||||
Assert.True(state.IsOnline); // liveness unchanged
|
||||
Assert.True(state.IsMetricsStale); // was impossible before the fix
|
||||
}
|
||||
|
||||
/// <summary>A heartbeat-only site inside the window is NOT falsely flagged.</summary>
|
||||
[Fact]
|
||||
public void HeartbeatOnlySite_WithinStaleWindow_IsNotFlagged()
|
||||
{
|
||||
var (aggregator, time) = NewAggregator(metricsStaleTimeout: TimeSpan.FromMinutes(2));
|
||||
aggregator.MarkHeartbeat("site-a", time.GetUtcNow());
|
||||
time.Advance(TimeSpan.FromMinutes(1));
|
||||
aggregator.MarkHeartbeat("site-a", time.GetUtcNow());
|
||||
aggregator.CheckForOfflineSites();
|
||||
Assert.False(aggregator.GetSiteState("site-a")!.IsMetricsStale);
|
||||
}
|
||||
```
|
||||
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests --filter "FullyQualifiedName~HeartbeatOnlySite"` — expect the first test to FAIL (`IsMetricsStale` stays false; the null guard skips the site).
|
||||
3. Implement:
|
||||
- `SiteHealthState`: add
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Gets the instant this site was first observed by the aggregator (via a
|
||||
/// report or a heartbeat); <c>null</c> only for states not built by the
|
||||
/// aggregator (hand-constructed fixtures). Anchors metrics-staleness for a
|
||||
/// site that has NEVER delivered a report (review 01 round-2 N3): with
|
||||
/// <see cref="LastReportReceivedAt"/> null the staleness check previously
|
||||
/// skipped the site forever. <see cref="LastHeartbeatAt"/> cannot anchor
|
||||
/// this — every heartbeat advances it.
|
||||
/// </summary>
|
||||
public DateTimeOffset? FirstSeenAt { get; init; }
|
||||
```
|
||||
|
||||
- `CentralHealthAggregator.ProcessReport` register branch (lines 44-54): add `FirstSeenAt = now,`. `MarkHeartbeat` register branch (lines 114-122): add `FirstSeenAt = receivedAt,`. (Update branches never touch it — `with` copies preserve it.)
|
||||
- `CheckForOfflineSites` staleness guard (lines 300-312): replace the `state.LastReportReceivedAt is { } lastReport` condition with an anchor fallback:
|
||||
|
||||
```csharp
|
||||
// Second signal: … (keep the existing comment, append:)
|
||||
// A site that has NEVER reported (heartbeat-only registration,
|
||||
// LastReportReceivedAt == null) anchors on FirstSeenAt instead —
|
||||
// otherwise a pipeline dead from first boot is never flagged (N3).
|
||||
var reportAnchor = state.LastReportReceivedAt ?? state.FirstSeenAt;
|
||||
if (reportAnchor is { } anchor
|
||||
&& now - anchor > _options.MetricsStaleTimeout
|
||||
&& !state.IsMetricsStale)
|
||||
{
|
||||
var stale = state with { IsMetricsStale = true };
|
||||
if (_siteStates.TryUpdate(kvp.Key, stale, state))
|
||||
{
|
||||
_logger.LogWarning(
|
||||
"Site {SiteId} metrics are stale — online (heartbeats) but no report for {Elapsed}s{NeverNote} (timeout: {Timeout}s)",
|
||||
state.SiteId, (now - anchor).TotalSeconds,
|
||||
state.LastReportReceivedAt is null ? " (never reported since first contact)" : string.Empty,
|
||||
_options.MetricsStaleTimeout.TotalSeconds);
|
||||
}
|
||||
// CAS loss ⇒ a fresh report/heartbeat swapped in ⇒ correct to skip.
|
||||
}
|
||||
```
|
||||
|
||||
- `Component-HealthMonitoring.md` "Metrics staleness" bullet (line ~47): append — "A site that has **never** delivered a report is anchored on its first-contact time (`FirstSeenAt`), so a report pipeline dead from first boot is also flagged after `MetricsStaleTimeout` (previously such a site showed online-with-no-metrics indefinitely)."
|
||||
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests` — full project green (the CAS/flap/stale suites are the regression net).
|
||||
5. Commit:
|
||||
```
|
||||
git add src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthState.cs src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthAggregator.cs docs/requirements/Component-HealthMonitoring.md tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/CentralHealthAggregatorTests.cs
|
||||
git commit -m "fix(health): flag metrics-stale for never-reported sites via FirstSeenAt anchor — null LastReportReceivedAt no longer skips the check (plan R2-01 T8)"
|
||||
```
|
||||
|
||||
### Task 9: Purge the stale "cluster leader" narration from `CentralHealthReportLoop`
|
||||
|
||||
**Classification:** trivial (comment-only)
|
||||
**Estimated implement time:** ~2 min
|
||||
**Parallelizable with:** Task 1, Task 2, Task 4, Task 5, Task 6, Task 7, Task 8, Task 10, Task 11
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthReportLoop.cs` (class doc line 12-14; sequence-seed comment lines 44-45)
|
||||
|
||||
Round-2 N4: the behavior is correct (line 93 gates on `IClusterNodeProvider.SelfIsPrimary` = oldest Up member since PLAN-01 T6), but the comments still narrate "cluster leader (Primary)" — re-teaching the exact leader/oldest conflation round-1 S2 eradicated.
|
||||
|
||||
1. Confirm scope first: `grep -in "leader" src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthReportLoop.cs` — expect exactly lines 12, 44, 45.
|
||||
2. Edit:
|
||||
- Class doc (lines 12-14): "Only the cluster leader (Primary) generates reports — the standby's aggregator catches up on failover when it becomes Primary and starts its own loop." → "Only the active node (Primary = the **oldest Up member**, i.e. the singleton host — `IClusterNodeProvider.SelfIsPrimary`, never Akka cluster leadership) generates reports; the standby's aggregator catches up on failover when it becomes the oldest and its loop starts reporting."
|
||||
- Sequence-seed comment (lines 44-45): "reports from a newly-elected central leader always sort after reports from any prior leader" → "reports from a node that has newly become active (oldest Up member) always sort after reports from any previously-active node".
|
||||
3. Verify: `grep -in "leader" src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthReportLoop.cs` — **0 hits**. Comment-only change: `dotnet build src/ZB.MOM.WW.ScadaBridge.HealthMonitoring` — succeeds.
|
||||
4. Commit:
|
||||
```
|
||||
git add src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthReportLoop.cs
|
||||
git commit -m "docs(health): CentralHealthReportLoop comments say oldest-member, not cluster leader — stop re-teaching the S2 conflation (plan R2-01 T9)"
|
||||
```
|
||||
|
||||
### Task 10: Generalize the registrar — `SingletonRegistrar` with an optional role scope
|
||||
|
||||
**Classification:** high-risk (actor lifecycle helper used by all central singletons)
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** Task 1, Task 2, Task 4, Task 5, Task 6, Task 7, Task 8, Task 9
|
||||
**Files:**
|
||||
- Modify (rename): `src/ZB.MOM.WW.ScadaBridge.Host/Actors/CentralSingletonRegistrar.cs` → `src/ZB.MOM.WW.ScadaBridge.Host/Actors/SingletonRegistrar.cs` (use `git mv`)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs` (rename at the block comment lines 459-465 and the 7 central call sites: lines 466, 499, 564, 609, 632, 663, 689)
|
||||
- Modify (rename): `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CentralSingletonRegistrarTests.cs` → `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SingletonRegistrarTests.cs`
|
||||
- Modify: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CoordinatedShutdownTests.cs` (line 56 source-grep string)
|
||||
|
||||
Round-2 N5 first half: `CentralSingletonRegistrar` cannot express `.WithRole(siteRole)`, which is the sole reason the two site singletons stay hand-rolled without drains (`AkkaHostedService.cs:464-465`). Add an optional `role` applied to BOTH the manager and proxy settings, and rename the class — "Central" becomes a lie the moment it hosts site singletons.
|
||||
|
||||
1. Write the failing test — in the (renamed) `SingletonRegistrarTests.cs`, rename the class + existing references `CentralSingletonRegistrar` → `SingletonRegistrar`, and add:
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task Start_WithRoleScope_CreatesRoleScopedSingleton_ThatAnswers()
|
||||
{
|
||||
// Round-2 N5: role scoping is what kept the two SITE singletons
|
||||
// (deployment-manager, event-log-handler) hand-rolled and drain-less.
|
||||
using var system = CreateSingleNodeClusterSystem(); // roles = ["Central"]
|
||||
var handle = SingletonRegistrar.Start(
|
||||
system, "role-widget", Props.Create(() => new EchoActor()),
|
||||
NullLogger.Instance, drainTimeout: TimeSpan.FromSeconds(2), role: "Central");
|
||||
|
||||
Assert.EndsWith("/user/role-widget-singleton", handle.Manager.Path.ToString());
|
||||
Assert.EndsWith("/user/role-widget-proxy", handle.Proxy.Path.ToString());
|
||||
|
||||
// The node holds the role => manager instantiates the singleton and
|
||||
// the role-scoped proxy resolves it.
|
||||
var echo = await handle.Proxy.Ask<string>("hi", TimeSpan.FromSeconds(20));
|
||||
Assert.Equal("hi", echo);
|
||||
}
|
||||
```
|
||||
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~SingletonRegistrarTests"` — expect FAIL (compile: no `SingletonRegistrar` type / no `role` parameter).
|
||||
3. Implement:
|
||||
- `git mv src/ZB.MOM.WW.ScadaBridge.Host/Actors/CentralSingletonRegistrar.cs src/ZB.MOM.WW.ScadaBridge.Host/Actors/SingletonRegistrar.cs`; rename the class; signature → `Start(ActorSystem system, string name, Props singletonProps, ILogger logger, TimeSpan? drainTimeout = null, string? role = null)`; apply the role to both settings:
|
||||
|
||||
```csharp
|
||||
var managerSettings = ClusterSingletonManagerSettings.Create(system).WithSingletonName(name);
|
||||
if (role is not null) managerSettings = managerSettings.WithRole(role);
|
||||
// … ClusterSingletonManager.Props(singletonProps, PoisonPill.Instance, managerSettings) …
|
||||
|
||||
var proxySettings = ClusterSingletonProxySettings.Create(system).WithSingletonName(name);
|
||||
if (role is not null) proxySettings = proxySettings.WithRole(role);
|
||||
// … ClusterSingletonProxy.Props($"/user/{name}-singleton", proxySettings) …
|
||||
```
|
||||
|
||||
Update the class XML doc: drop "central"-only framing; drain rationale now reads "so in-flight EF (central) or SQLite (site) work completes before handover"; note the optional `role` maps to `ClusterSingletonManagerSettings/ProxySettings.WithRole` (review 01 round-2 N5).
|
||||
- `AkkaHostedService.cs`: mechanical rename at the 7 call sites (466, 499, 564, 609, 632, 663, 689) and rewrite the lines-459-465 comment's last sentence ("The two SITE singletons stay hand-rolled below because they are role-scoped") → "Site singletons use the same registrar with `role: siteRole` (Task 11 / round-2 N5)." (Task 11 lands that; wording here is forward-referencing but true within the same plan.)
|
||||
- `git mv tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CentralSingletonRegistrarTests.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SingletonRegistrarTests.cs`.
|
||||
- `CoordinatedShutdownTests.cs` line 56: `Assert.Contains("CentralSingletonRegistrar.Start(", content);` → `Assert.Contains("SingletonRegistrar.Start(", content);` (also update the comment above it).
|
||||
- Sanity: `grep -rn "CentralSingletonRegistrar" src/ tests/ --include="*.cs"` — 0 hits.
|
||||
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~SingletonRegistrarTests|FullyQualifiedName~CoordinatedShutdownTests"` — PASS. Then `dotnet build ZB.MOM.WW.ScadaBridge.slnx` — clean.
|
||||
5. Commit:
|
||||
```
|
||||
git add src/ZB.MOM.WW.ScadaBridge.Host/Actors/SingletonRegistrar.cs src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SingletonRegistrarTests.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CoordinatedShutdownTests.cs
|
||||
git rm --cached src/ZB.MOM.WW.ScadaBridge.Host/Actors/CentralSingletonRegistrar.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CentralSingletonRegistrarTests.cs 2>/dev/null || true
|
||||
git commit -m "refactor(host): CentralSingletonRegistrar -> SingletonRegistrar with optional role scope — unblocks site-singleton drains (plan R2-01 T10)"
|
||||
```
|
||||
(`git mv` already stages the renames; the `git rm --cached … || true` line is a no-op safety net if the moves were done as delete+create.)
|
||||
|
||||
### Task 11: Route the two site singletons through the registrar — deployment-manager + event-log-handler gain drains
|
||||
|
||||
**Classification:** high-risk (site actor wiring refactor)
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** Task 1, Task 2, Task 4, Task 5, Task 6, Task 7, Task 8, Task 9
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs` (site branch: deployment-manager block lines ~791-820; event-log-handler block lines ~836-856)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CoordinatedShutdownTests.cs`
|
||||
|
||||
Round-2 N5 second half: `deployment-manager` and `event-log-handler` are hand-rolled with bare `PoisonPill` termination and no `PhaseClusterLeave` drain. The round-1 S5 rationale applies identically on the site side: the DeploymentManager's SQLite writes (static overrides, `native_alarm_state`) should drain before handover — currently safe-but-lossy on graceful failover.
|
||||
|
||||
1. Write the failing test — in `CoordinatedShutdownTests.cs`, update the existing `AllCentralSingletons_RegisterThroughRegistrarWithDrain` (lines 45-70): change the final assertion `Assert.Equal(2, CountOccurrences(content, "ClusterSingletonManager.Props("));` → `Assert.Equal(0, …)` and replace its "the only two left are the SITE singletons" comment; then add:
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public void SiteSingletons_RegisterThroughRegistrarWithDrain()
|
||||
{
|
||||
var hostProjectDir = FindHostProjectDirectory();
|
||||
Assert.NotNull(hostProjectDir);
|
||||
var content = File.ReadAllText(Path.Combine(hostProjectDir!, "Actors", "AkkaHostedService.cs"));
|
||||
|
||||
// Round-2 N5: the two role-scoped SITE singletons previously stayed
|
||||
// hand-rolled (bare PoisonPill, no PhaseClusterLeave drain). They now
|
||||
// go through SingletonRegistrar.Start(..., role: siteRole), which
|
||||
// always adds the GracefulStop drain — in-flight SQLite writes
|
||||
// (static overrides, native_alarm_state) complete before handover.
|
||||
Assert.Contains("\"deployment-manager\"", content);
|
||||
Assert.Contains("\"event-log-handler\"", content);
|
||||
Assert.Contains("role: siteRole", content);
|
||||
Assert.Equal(0, CountOccurrences(content, "ClusterSingletonManager.Props("));
|
||||
}
|
||||
```
|
||||
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~SiteSingletons_RegisterThroughRegistrarWithDrain"` — expect FAIL (`ClusterSingletonManager.Props(` count is 2; no `role: siteRole`).
|
||||
3. Implement in the site branch of `AkkaHostedService.cs`:
|
||||
- Deployment-manager block (~791-820): replace the hand-rolled `ClusterSingletonManager.Props`/`ClusterSingletonProxy.Props` pair with
|
||||
|
||||
```csharp
|
||||
// Deployment Manager — role-scoped singleton via SingletonRegistrar
|
||||
// (review 01 round-2 N5): previously hand-rolled with bare PoisonPill
|
||||
// termination and NO PhaseClusterLeave drain, so in-flight SQLite
|
||||
// writes (static overrides, native_alarm_state) could be cut off on
|
||||
// graceful failover. Names unchanged: deployment-manager-singleton /
|
||||
// deployment-manager-proxy.
|
||||
var dm = SingletonRegistrar.Start(
|
||||
_actorSystem!, "deployment-manager",
|
||||
Props.Create(() => new DeploymentManagerActor(
|
||||
storage, compilationService, sharedScriptLibrary, streamManager,
|
||||
siteRuntimeOptionsValue, dmLogger, dclManager, replicationActor,
|
||||
siteHealthCollector, _serviceProvider, null, deploymentConfigFetcher)),
|
||||
_logger, role: siteRole);
|
||||
var dmProxy = dm.Proxy;
|
||||
```
|
||||
|
||||
(Keep the `dmProxy` variable name — everything downstream, `SiteCommunicationActor` construction and `RegisterLocalHandler(LocalHandlerType.Artifacts, dmProxy)`, is untouched.)
|
||||
- Event-log-handler block (~836-856): inside the existing `if (eventLogQueryService != null)`:
|
||||
|
||||
```csharp
|
||||
var eventLog = SingletonRegistrar.Start(
|
||||
_actorSystem, "event-log-handler",
|
||||
Props.Create(() => new SiteEventLogging.EventLogHandlerActor(eventLogQueryService)),
|
||||
_logger, role: siteRole);
|
||||
siteCommActor.Tell(new RegisterLocalHandler(LocalHandlerType.EventLog, eventLog.Proxy));
|
||||
```
|
||||
|
||||
(Keep the existing "Event log handler — cluster singleton so queries always reach the active node…" comment above it.)
|
||||
- Actor/singleton names are reproduced exactly by the registrar (`deployment-manager-singleton`/`-proxy`, `event-log-handler-singleton`/`-proxy`), so path-pinning tests and the ClusterClient receptionist wiring see no change.
|
||||
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests` (full project — path/health-check/audit-wiring suites are the regression net) and `dotnet build ZB.MOM.WW.ScadaBridge.slnx` — PASS/clean.
|
||||
5. Commit:
|
||||
```
|
||||
git add src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CoordinatedShutdownTests.cs
|
||||
git commit -m "fix(host): site singletons (deployment-manager, event-log-handler) via SingletonRegistrar — PhaseClusterLeave drains for site SQLite writes (plan R2-01 T11)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Dependencies on other plans
|
||||
|
||||
- **PLAN-R2-02 owns** the oldest-vs-leader predicate unification inside `SiteCommunicationActor`/`SiteReplicationActor` and everything resync-related. This plan does not touch those files; the coverage table references it.
|
||||
- **PLAN-R2-08 owns** N6 (folding the two live PLAN-01 deferrals — the keep-oldest active-crash gap and, post-T5/T6, the "sync the artifact to the host" residue — into `docs/plans/2026-07-08-deferred-work-register.md`). Task 7 here only corrects the factually wrong tracker prose; it adds no register rows.
|
||||
- The **SBR keep-oldest active-crash availability gap stays a registered deferred USER decision** — no task here changes the strategy/topology; Tasks 1-3 make the gap observable and measured, which is the input that decision needs.
|
||||
|
||||
## Execution order
|
||||
|
||||
**Drill spine (serial):** Task 1 (script) → Task 2 (docs) → Task 3 (live run, needs the docker cluster exclusively).
|
||||
|
||||
**Registrar spine (serial):** Task 10 → Task 11.
|
||||
|
||||
**Freely parallel at any point:** Tasks 4, 5, 6, 8, 9 (all file-disjoint); Task 7 after 5+6.
|
||||
|
||||
**Highest value if time-boxed:** Task 3 (the never-run drill, run at last) + Task 11 (real durability gap) + Task 8 (real monitoring blind spot).
|
||||
|
||||
---
|
||||
|
||||
## Findings Coverage
|
||||
|
||||
Every round-2 finding (N1-N6), the round-1 partials the report says warrant completion, and the cross-plan rulings:
|
||||
|
||||
| # | Round-2 finding | Severity | Task(s) / Disposition |
|
||||
|---|----------------|----------|----------------------|
|
||||
| N1 | Failover drill kills the wrong (active/oldest) node; first-seed bootstrap gap means the survivor's recovery story doesn't converge; drill never executed; ~25s promise undeliverable in its most likely case | High | Tasks 1 (drill rewrite: standby-victim default + explicit active-gap mode), 2 (narrative + first-seed constraint **documented** with operator actions — deliberately NOT reordered to self-first seeds: simultaneous cold-start self-joins would risk two independent clusters), 3 (RUN the drill live, both directions, record measured timings), 4 (wire/unskip `FailoverTimingTests` to `TwoNodeClusterFixture`, measure the ~25s envelope at production timings — **also covers report-08 NF2 by reference**). The keep-oldest topology/strategy remedy itself **stays deferred: registered USER decision** (master tracker 2026-07-08) — Tasks 1-3 document/measure it, never fix it |
|
||||
| N2 | wonder-app-vd03 overlay edits unapplied; deferral rationale factually wrong (artifact IS in the working tree, merely gitignored) | Medium | Tasks 5 (NodeName + AllowSingleNodeCluster + drop phantom seeds, both overlays), 6 (`sc.exe failureflag` completing the recovery actions + RUNBOOK step — note: `sc.exe failure` itself is already on disk, contra the report), 7 (correct the deferral record in the master tracker). No git commits for the gitignored artifact itself |
|
||||
| N3 | `IsMetricsStale` can never fire for a site that has never delivered a report (null `LastReportReceivedAt` guard) | Low | Task 8 (`FirstSeenAt` anchor; no UI change needed — badge already keys on the flag) |
|
||||
| N4 | `CentralHealthReportLoop` still narrates "cluster leader (Primary)" | Low | Task 9 |
|
||||
| N5 | Site-role singletons lack drain tasks; registrar can't express role scoping | Low | Tasks 10 (role param + rename to `SingletonRegistrar`), 11 (route `deployment-manager` + `event-log-handler` through it) |
|
||||
| N6 | PLAN-01's two live deferrals missing from the canonical deferred-work register | Low | **Reference → PLAN-R2-08** (register consolidation is its scope). Task 7 here only fixes the falsehood in the master-tracker prose |
|
||||
| S12 (R1 partial) | Single-node artifact still ships the phantom seed, lacks `AllowSingleNodeCluster` | — | Task 5 (the artifact half; the flag/validator half shipped in PLAN-01 T16) |
|
||||
| U2 (R1 partial) | install.ps1 recovery actions deferred; recovery contract doesn't converge for a non-first-seed survivor | — | Task 6 (failureflag) + Task 2 (first-seed constraint + operator actions in the spec'd recovery contract) |
|
||||
| U5 (R1 not fixed) | `NodeName` missing from wonder-app-vd03 overlays → NULL `SourceNode` | — | Task 5 |
|
||||
| C3 (R1 partial) | LDAP `Transport: None, AllowInsecure: true` remains in the deploy artifact | — | **Stays accepted posture per the report** — bundled loopback GLAuth with documented AD-repoint instructions; a future over-the-wire AD repoint must revisit transport. No task |
|
||||
| — | Oldest-vs-leader predicate unification in `SiteCommunicationActor`/`SiteReplicationActor` + resync work | — | **Reference → PLAN-R2-02** (explicit cross-plan ownership; not touched here) |
|
||||
@@ -0,0 +1,82 @@
|
||||
{
|
||||
"planPath": "archreview/plans/PLAN-R2-01-cluster-host-failover.md",
|
||||
"tasks": [
|
||||
{
|
||||
"id": 1,
|
||||
"subject": "Task 1: Rewrite failover-drill.sh — standby-victim default + explicit active-victim gap mode",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"subject": "Task 2: Correct the recovery narrative + document the first-seed bootstrap constraint",
|
||||
"status": "pending",
|
||||
"blockedBy": [
|
||||
1
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"subject": "Task 3: RUN the drill live (both directions) and record measured timings",
|
||||
"status": "pending",
|
||||
"blockedBy": [
|
||||
1,
|
||||
2
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"subject": "Task 4: Wire FailoverTimingTests to TwoNodeClusterFixture — measure the ~25s envelope in-process",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"subject": "Task 5: Apply the wonder-app-vd03 appsettings overlay edits (NodeName + AllowSingleNodeCluster, drop phantom seeds)",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"subject": "Task 6: Complete the install.ps1 recovery actions (sc.exe failureflag) + RUNBOOK recovery step",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
"subject": "Task 7: Correct the factually wrong N2 deferral record in the master tracker",
|
||||
"status": "pending",
|
||||
"blockedBy": [
|
||||
5,
|
||||
6
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 8,
|
||||
"subject": "Task 8: Metrics-staleness for never-reported sites (FirstSeenAt anchor)",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": 9,
|
||||
"subject": "Task 9: Purge the stale \"cluster leader\" narration from CentralHealthReportLoop",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": 10,
|
||||
"subject": "Task 10: Generalize the registrar — SingletonRegistrar with an optional role scope",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": 11,
|
||||
"subject": "Task 11: Route the two site singletons through the registrar — deployment-manager + event-log-handler gain drains",
|
||||
"status": "pending",
|
||||
"blockedBy": [
|
||||
10
|
||||
]
|
||||
}
|
||||
],
|
||||
"lastUpdated": "2026-07-13T03:25:44Z"
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,115 @@
|
||||
{
|
||||
"planPath": "archreview/plans/PLAN-R2-02-communication-store-and-forward.md",
|
||||
"tasks": [
|
||||
{
|
||||
"id": 1,
|
||||
"subject": "Task 1: Shared oldest-Up active-node evaluator, reachable from Communication and SiteRuntime",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"subject": "Task 2: Two-node divergence repro — leader≠oldest wipes the delivering node's buffer (failing first)",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"subject": "Task 3: Swap SiteReplicationActor to the shared oldest-Up predicate + Host wires the delivery-gate delegate",
|
||||
"status": "pending",
|
||||
"blockedBy": [
|
||||
1,
|
||||
2
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"subject": "Task 4: Swap SiteCommunicationActor.DefaultIsActiveCheck + Host wiring + doc sync",
|
||||
"status": "pending",
|
||||
"blockedBy": [
|
||||
3
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"subject": "Task 5: Chunked resync protocol — additive messages, byte-budgeted chunker, active-side chunked answer",
|
||||
"status": "pending",
|
||||
"blockedBy": [
|
||||
3
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"subject": "Task 6: Standby-side chunk assembly, atomic apply, ack + the N5 race comment",
|
||||
"status": "pending",
|
||||
"blockedBy": [
|
||||
5
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
"subject": "Task 7: Resync delivery confirmation on the active node + telemetry + doc rewrite",
|
||||
"status": "pending",
|
||||
"blockedBy": [
|
||||
6
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 8,
|
||||
"subject": "Task 8: Publish _sweepTask only when the sweep CAS is won (unclobber the shutdown drain)",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": 9,
|
||||
"subject": "Task 9: Eager validation for SweepBatchLimit / SweepTargetParallelism",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": 10,
|
||||
"subject": "Task 10: Coalesce live-alarm delta publishes (bound the per-circuit fan-out) [MUTEX with PLAN-R2-07]",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": 11,
|
||||
"subject": "Task 11: Aggregator lifecycle — queued re-seed after reconnect + stream-generation stamp",
|
||||
"status": "pending",
|
||||
"blockedBy": [
|
||||
10
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 12,
|
||||
"subject": "Task 12: Instance-injected ReconnectDelay/StabilityWindow (kill the process-global test seams)",
|
||||
"status": "pending",
|
||||
"blockedBy": [
|
||||
11
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 13,
|
||||
"subject": "Task 13: Production caller for RemoveSiteAsync — dispose a deleted site's gRPC channels",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": 14,
|
||||
"subject": "Task 14: Documented acceptance — standby-node aggregators + deleted-site viewer behavior",
|
||||
"status": "pending",
|
||||
"blockedBy": [
|
||||
4
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 15,
|
||||
"subject": "Task 15: Single-endpoint sites can go live [MUTEX with PLAN-R2-07]",
|
||||
"status": "pending",
|
||||
"blockedBy": [
|
||||
12
|
||||
]
|
||||
}
|
||||
],
|
||||
"lastUpdated": "2026-07-13T03:28:50Z"
|
||||
}
|
||||
@@ -0,0 +1,485 @@
|
||||
# PLAN-R2-03 — Site Runtime & DCL Round-2 Hardening Implementation Plan
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
|
||||
|
||||
**Goal:** Fix the round-2 findings in `archreview/03-site-runtime-dcl.md` (dated 2026-07-12) — the one Medium residual on the MxGateway reconnect path (a stale event loop's non-OCE fault can still flap the fresh connection through the generic catch) plus five Low polish items: missing failure mapping on the expression-eval `PipeTo`, two PLAN-03 option knobs absent from the eager options validator, the deploy-path double-compile (adopt the earmarked compile cache), the reintroduced "off-thread" spec-drift sentence, and the undocumented Expression-trigger/script-scheduler coupling. Round-1 deferrals P5 and P6 stay deferred (coverage-table rows only; Tasks 5/6's compile cache shrinks P6's warm-process cost as a side effect).
|
||||
|
||||
**Architecture:** All changes are site-side, inside `ZB.MOM.WW.ScadaBridge.SiteRuntime` and `ZB.MOM.WW.ScadaBridge.DataConnectionLayer`, plus their design doc. The established discipline is preserved: adapter lifecycle faults are classified on the *loop's own* cancellation token (never the mutable fields a newer generation may have replaced), every off-thread result — including a *faulted* task — returns to the actor via `PipeTo` and is applied on the actor thread, and options fail fast at boot with key-naming messages per the PLAN-08 §1.5 convention. The N4 compile cache mirrors TemplateEngine's `ScriptCompileVerdictCache` (SHA-256-keyed, bounded, clear-on-overflow) but is site-local and stores the full `ScriptCompilationResult` — sound because Roslyn `Script<T>` objects are immutable and thread-safe and the result's error text is name-free — so the synchronous deploy gate (deliberately on the singleton mailbox, see N5) and the `InstanceActor.PreStart` recompile share one Roslyn compile per script body per process lifetime. Docs travel with code.
|
||||
|
||||
**Tech Stack:** C#/.NET, Akka.NET (ReceiveActor, TestKit.Xunit2), Roslyn scripting, gRPC (Grpc.Core exception shapes via the `ZB.MOM.WW.MxGateway.Client` package), xunit + NSubstitute.
|
||||
|
||||
Build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Test per-project: `dotnet test tests/<project> --filter <name>`.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: MxGatewayDataConnection — a stale event loop's fault on a cancelled token must not signal Disconnected
|
||||
|
||||
**Classification:** high-risk (adapter reconnect lifecycle, concurrency with the dying loop)
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 2, 3, 4, 5
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/MxGatewayDataConnection.cs` (`ConnectAsync` ~lines 106–108, `RunEventLoopAsync` ~lines 111–130)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Adapters/MxGatewayDataConnectionReconnectTests.cs`
|
||||
|
||||
The S1 fix cancels the previous loop's CTS and disposes the previous client (`:77–89`), then resets `_disconnectFired` (`:91`) — but `Cancel()` is not synchronous with the loop's exit. The old loop's `await foreach` over the gRPC stream surfaces its failure milliseconds later, and a cancelled/disposed Grpc.Net stream commonly faults with `RpcException(StatusCode.Cancelled)` (the default without `ThrowOperationCanceledOnCancellation`) or `ObjectDisposedException` from the concurrent `DisposeAsync` — neither is an `OperationCanceledException`, so both fall into the generic `catch` (`:125–129`) → `RaiseDisconnected()` against the *reset* guard → the exact S1 scenario-2 flap, potentially self-sustaining.
|
||||
|
||||
1. Add failing tests to the existing reconnect test class (extend its per-client substitute fixture so each client's `RunEventLoopAsync` returns a controllable `TaskCompletionSource` task instead of `Task.Delay(Timeout.Infinite)`):
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task StaleEventLoopRpcFault_AfterReconnect_DoesNotSignalDisconnected()
|
||||
{
|
||||
var clients = new List<IMxGatewayClient>();
|
||||
var loops = new List<TaskCompletionSource>();
|
||||
var loopTokens = new List<CancellationToken>();
|
||||
var factory = Substitute.For<IMxGatewayClientFactory>();
|
||||
factory.Create().Returns(_ =>
|
||||
{
|
||||
var c = Substitute.For<IMxGatewayClient>();
|
||||
var tcs = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||
loops.Add(tcs);
|
||||
c.RunEventLoopAsync(Arg.Any<Action<MxValueUpdate>>(), Arg.Do<CancellationToken>(t => loopTokens.Add(t)))
|
||||
.Returns(tcs.Task);
|
||||
clients.Add(c);
|
||||
return c;
|
||||
});
|
||||
var adapter = new MxGatewayDataConnection(factory, NullLogger<MxGatewayDataConnection>.Instance);
|
||||
var disconnects = 0;
|
||||
adapter.Disconnected += () => Interlocked.Increment(ref disconnects);
|
||||
var details = new Dictionary<string, string> { ["Endpoint"] = "http://gw:5000", ["ApiKey"] = "k" };
|
||||
|
||||
await adapter.ConnectAsync(details);
|
||||
await WaitUntilAsync(() => loopTokens.Count == 1);
|
||||
await adapter.ConnectAsync(details); // reconnect: cancels loop #0, resets _disconnectFired
|
||||
await WaitUntilAsync(() => loopTokens.Count == 2);
|
||||
|
||||
// The OLD loop observes its cancellation as a gRPC fault, not an OCE —
|
||||
// the Grpc.Net default without ThrowOperationCanceledOnCancellation.
|
||||
loops[0].SetException(new RpcException(new Status(StatusCode.Cancelled, "call cancelled")));
|
||||
await Task.Delay(200);
|
||||
Assert.Equal(0, disconnects); // stale fault absorbed — the fresh connection must not flap
|
||||
|
||||
// The CURRENT loop's genuine fault must still signal, exactly once.
|
||||
loops[1].SetException(new RpcException(new Status(StatusCode.Unavailable, "gateway gone")));
|
||||
await WaitUntilAsync(() => disconnects == 1);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task StaleEventLoopObjectDisposedFault_AfterReconnect_DoesNotSignalDisconnected()
|
||||
{
|
||||
// Identical drive, but the old loop faults with the concurrent-DisposeAsync shape.
|
||||
// … same fixture …
|
||||
loops[0].SetException(new ObjectDisposedException("MxGatewayClient"));
|
||||
await Task.Delay(200);
|
||||
Assert.Equal(0, disconnects);
|
||||
}
|
||||
```
|
||||
|
||||
(`RpcException`/`Status`/`StatusCode` come from `Grpc.Core`, transitively via the `ZB.MOM.WW.MxGateway.Client` package the test project already references — `RealMxGatewayClient.cs:227` catches `RpcException` in the main project. If the test assembly cannot resolve them, add a `Grpc.Net.Client` PackageReference; the version is already pinned in `Directory.Packages.props`.)
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~StaleEventLoopRpcFault_AfterReconnect|FullyQualifiedName~StaleEventLoopObjectDisposedFault_AfterReconnect"` → expect **FAIL** (the generic catch raises `Disconnected` against the reset guard: `disconnects == 1` after the stale fault).
|
||||
3. Implement:
|
||||
- Change `RunEventLoopAsync` to take the client AND token as parameters — `private async Task RunEventLoopAsync(IMxGatewayClient client, CancellationToken ct)` — and use `client.RunEventLoopAsync(…)` instead of the `_client!` field read (the old loop must never bind a newer generation's client).
|
||||
- Add a filtered catch **before** the generic one:
|
||||
|
||||
```csharp
|
||||
catch (Exception ex) when (ct.IsCancellationRequested)
|
||||
{
|
||||
// Teardown fault of a superseded loop (N1): Cancel() is not synchronous with
|
||||
// the loop's exit — a cancelled/disposed gRPC stream commonly surfaces as
|
||||
// RpcException(StatusCode.Cancelled) (Grpc.Net default without
|
||||
// ThrowOperationCanceledOnCancellation) or ObjectDisposedException from the
|
||||
// concurrent DisposeAsync, milliseconds AFTER ConnectAsync has already reset
|
||||
// _disconnectFired. Any fault on a cancelled token is normal shutdown;
|
||||
// raising Disconnected here would flap the NEW healthy session.
|
||||
_logger.LogDebug(ex,
|
||||
"MxGateway event loop faulted after cancellation — superseded loop teardown; not signalling disconnect");
|
||||
}
|
||||
```
|
||||
|
||||
- In `ConnectAsync`, capture the token and client into locals before the lambda (a rapid double-reconnect must not hand loop #1 loop #2's token/client — the current `Task.Run(() => RunEventLoopAsync(_eventLoopCts.Token))` reads the *field* lazily):
|
||||
|
||||
```csharp
|
||||
_eventLoopCts = new CancellationTokenSource();
|
||||
var loopToken = _eventLoopCts.Token; // capture NOW: the field may be replaced by a
|
||||
var loopClient = _client; // subsequent reconnect before Task.Run executes (N1)
|
||||
_ = Task.Run(() => RunEventLoopAsync(loopClient, loopToken));
|
||||
```
|
||||
|
||||
4. Run the two new tests → **PASS**; then the full adapter suite: `dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~MxGatewayDataConnection"` → all green (the existing `Reconnect_DisposesPreviousClient_AndCancelsPreviousEventLoop` test pins the S1 behavior and must survive the signature change).
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/MxGatewayDataConnection.cs tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Adapters/MxGatewayDataConnectionReconnectTests.cs && git commit -m "fix(dcl): stale MxGateway event-loop fault on a cancelled token no longer flaps the fresh connection (plan R2-03 T1)"`
|
||||
|
||||
### Task 2: ScriptActor — failure mapping on the expression-eval `PipeTo` (a faulted scheduler task must not kill the trigger)
|
||||
|
||||
**Classification:** high-risk (actor concurrency, trigger liveness)
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1, 3, 4, 5, 6
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptActor.cs` (Receive block ~line 161, `StartExpressionEvaluation` PipeTo ~lines 311–313, internal messages ~line 623)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/ScriptActorTests.cs`
|
||||
|
||||
`StartExpressionEvaluation` pipes with `success:` only (`:312–313`). The inner body catches everything, so the only fault source is the outer `Task.Factory.StartNew` itself (e.g. `ObjectDisposedException` from a disposed `ScriptExecutionScheduler` during shutdown/test teardown) — but if it ever fires, the actor gets an unhandled `Status.Failure` and `_evalInFlight` stays `true` forever (`:284–285`): every subsequent change parks in `_evalPending` and the expression trigger is dead for the actor's lifetime with no log.
|
||||
|
||||
1. Failing test. Add a public static gate hook to the test file (the `CompileRawTriggerExpression` helper compiles *without* the trust validator, so the expression may reference it — same trick as the P1 thread-name tests):
|
||||
|
||||
```csharp
|
||||
public static class EvalGate
|
||||
{
|
||||
public static readonly SemaphoreSlim Gate = new(0);
|
||||
private static int _entries;
|
||||
public static int Entries => Volatile.Read(ref _entries);
|
||||
public static bool Block() { Interlocked.Increment(ref _entries); Gate.Wait(); return false; }
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ExpressionEvalTaskFault_ClearsInFlight_AndDrainsPendingEvaluation()
|
||||
{
|
||||
var expr = CompileRawTriggerExpression(
|
||||
"ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors.EvalGate.Block()");
|
||||
var (actor, _) = CreateTriggeredActor(
|
||||
"ExprFault", "Expression", "{\"expression\":\"true\",\"mode\":\"OnTrue\"}", null, expr);
|
||||
try
|
||||
{
|
||||
actor.Tell(Change("A", "1")); // eval starts on the scheduler and BLOCKS → _evalInFlight = true
|
||||
AwaitAssert(() => Assert.Equal(1, EvalGate.Entries), TimeSpan.FromSeconds(10));
|
||||
actor.Tell(Change("A", "2")); // coalesces → _evalPending = true
|
||||
|
||||
// Simulate the faulted scheduler task the PipeTo failure mapping now surfaces
|
||||
// (e.g. ObjectDisposedException from a disposed ScriptExecutionScheduler).
|
||||
actor.Tell(new ScriptActor.ExpressionEvalFailed(new ObjectDisposedException("scheduler")));
|
||||
|
||||
// In-flight cleared + pending drained ⇒ a SECOND evaluation starts and blocks too.
|
||||
AwaitAssert(() => Assert.Equal(2, EvalGate.Entries), TimeSpan.FromSeconds(10));
|
||||
}
|
||||
finally
|
||||
{
|
||||
EvalGate.Gate.Release(EvalGate.Entries); // ALWAYS free the shared scheduler threads
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
(Note the expression must reference the gate's fully-qualified name; the raw compile references `typeof(object).Assembly` only, so also add the test assembly to `CompileRawTriggerExpression`'s `.WithReferences(...)` for this test — or add an overload taking extra references. The `finally` release is mandatory: the gate blocks threads of the *process-wide* `ScriptExecutionScheduler.Shared`.)
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ExpressionEvalTaskFault_ClearsInFlight"` → expect **FAIL** — the RED step is the compile failure (`ScriptActor.ExpressionEvalFailed` does not exist); behaviorally, without the fix the second evaluation never starts either.
|
||||
3. Implement:
|
||||
- New message next to `ExpressionEvalResult` (`:623`) — `internal` (not `private`) so the TestKit can Tell it, same visibility idiom as `IntervalTick`/`ScriptExecutionCompleted`:
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Piped back to self when the off-dispatcher evaluation TASK itself faults
|
||||
/// (e.g. ObjectDisposedException from a disposed ScriptExecutionScheduler
|
||||
/// during shutdown) — the inner body's catch never sees that. Without this
|
||||
/// mapping the actor receives an unhandled Status.Failure and _evalInFlight
|
||||
/// stays true forever, permanently parking the expression trigger (N2).
|
||||
/// </summary>
|
||||
internal sealed record ExpressionEvalFailed(Exception Cause);
|
||||
```
|
||||
|
||||
- Ctor Receive block (after the `ExpressionEvalResult` registration at `:161`): `Receive<ExpressionEvalFailed>(HandleExpressionEvalFailed);`
|
||||
- PipeTo (`:312–313`): `….PipeTo(self, success: r => new ExpressionEvalResult(r), failure: ex => new ExpressionEvalFailed(ex));`
|
||||
- Handler — log via the existing `LogExpressionError` (`:397`, already increments the health counter), then reuse the result path with `false` (consistent with the inner catch's treated-as-false semantics — clears in-flight, applies the false edge, drains pending):
|
||||
|
||||
```csharp
|
||||
private void HandleExpressionEvalFailed(ExpressionEvalFailed msg)
|
||||
{
|
||||
LogExpressionError(msg.Cause);
|
||||
HandleExpressionEvalResult(new ExpressionEvalResult(false));
|
||||
}
|
||||
```
|
||||
|
||||
4. Run the new test → **PASS**; then `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ScriptActorTests"` → all green (the P1 expression/WhileTrue suites are the regression canary).
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/ScriptActorTests.cs && git commit -m "fix(siteruntime): faulted expression-eval task clears _evalInFlight instead of killing the ScriptActor trigger (plan R2-03 T2)"`
|
||||
|
||||
### Task 3: AlarmActor — same failure mapping on the expression-eval `PipeTo`
|
||||
|
||||
**Classification:** high-risk (alarm state machine liveness)
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** 1, 2, 4, 5, 6
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/AlarmActor.cs` (Receive block ~line 188, `StartExpressionEvaluation` PipeTo ~lines 552–554, internal messages ~line 797)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/AlarmActorTests.cs`
|
||||
|
||||
Structurally identical to Task 2 — `AlarmActor.cs:552–554` pipes success-only, `:523–524` sets `_evalInFlight`, so a faulted task permanently kills the Expression alarm.
|
||||
|
||||
1. Failing test (reuse the `ExpressionAlarm_EvaluatesOnSchedulerThread_AndActivates` harness in `AlarmActorTests.cs:1031` — same `ResolvedAlarm`/`instanceProbe` construction):
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public void ExpressionEvalTaskFault_DoesNotKillTheAlarmTrigger()
|
||||
{
|
||||
var expr = CompileRawTriggerExpression("true");
|
||||
var alarmConfig = new ResolvedAlarm
|
||||
{
|
||||
CanonicalName = "ExprFaultAlarm",
|
||||
TriggerType = "Expression",
|
||||
TriggerConfiguration = "{\"expression\":\"true\"}",
|
||||
PriorityLevel = 1
|
||||
};
|
||||
var instanceProbe = CreateTestProbe();
|
||||
var alarm = ActorOf(Props.Create(() => new AlarmActor(
|
||||
"ExprFaultAlarm", "Pump1", instanceProbe.Ref, alarmConfig,
|
||||
null, _sharedLibrary, _options,
|
||||
NullLogger<AlarmActor>.Instance, expr)));
|
||||
|
||||
// Simulate the faulted scheduler task: must be handled (logged, in-flight
|
||||
// cleared, false applied) — NOT an unhandled Status.Failure.
|
||||
alarm.Tell(new AlarmActor.ExpressionEvalFailed(new ObjectDisposedException("scheduler")));
|
||||
|
||||
// The trigger must still be alive: the next change evaluates and activates.
|
||||
alarm.Tell(new AttributeValueChanged("Pump1", "A", "A", 1, "Good", DateTimeOffset.UtcNow));
|
||||
var change = instanceProbe.ExpectMsg<AlarmStateChanged>(TimeSpan.FromSeconds(10));
|
||||
Assert.Equal(AlarmState.Active, change.State);
|
||||
}
|
||||
```
|
||||
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ExpressionEvalTaskFault_DoesNotKillTheAlarmTrigger"` → expect **FAIL** (compile error — `AlarmActor.ExpressionEvalFailed` does not exist).
|
||||
3. Implement, mirroring Task 2:
|
||||
- `internal sealed record ExpressionEvalFailed(Exception Cause);` next to the existing `ExpressionEvalResult` (`:797`), with the same N2 XML doc.
|
||||
- `Receive<ExpressionEvalFailed>(HandleExpressionEvalFailed);` after `:188`.
|
||||
- PipeTo (`:553–554`) gains `failure: ex => new ExpressionEvalFailed(ex)`.
|
||||
- Handler — same shape as the inner catch's error path, then delegate to the false result (the class doc already promises "a throwing … expression is treated as false so that the state machine still runs — an Active alarm correctly clears"):
|
||||
|
||||
```csharp
|
||||
private void HandleExpressionEvalFailed(ExpressionEvalFailed msg)
|
||||
{
|
||||
_healthCollector?.IncrementAlarmError();
|
||||
_logger.LogError(msg.Cause,
|
||||
"Alarm {Alarm} trigger-expression evaluation task faulted on {Instance}; treated as false",
|
||||
_alarmName, _instanceName);
|
||||
HandleExpressionEvalResult(new ExpressionEvalResult(false));
|
||||
}
|
||||
```
|
||||
|
||||
4. Run the new test → **PASS**; then `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~AlarmActorTests"` → all green.
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/AlarmActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/AlarmActorTests.cs && git commit -m "fix(siteruntime): faulted expression-eval task clears _evalInFlight instead of killing the AlarmActor trigger (plan R2-03 T3)"`
|
||||
|
||||
### Task 4: Eagerly validate `TagSubscribeRetryIntervalMs` and `StuckScriptGraceMs`
|
||||
|
||||
**Classification:** small (additive validator rules per the PLAN-08 §1.5 convention)
|
||||
**Estimated implement time:** ~3 min
|
||||
**Parallelizable with:** 1, 2, 3, 5, 6
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptionsValidator.cs` (append after the `ConfigFetchRetryCount` rule, lines 56–58)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/SiteRuntimeOptionsValidatorTests.cs`
|
||||
|
||||
The validator covers ten options but not the two PLAN-03 added: `TagSubscribeRetryIntervalMs` (`SiteRuntimeOptions.cs:76` — a 0 hot-loops the subscribe retry, a negative value throws inside `InstanceActor.SendTagSubscribe`'s `ScheduleTellOnceCancelable`) and `StuckScriptGraceMs` (`SiteRuntimeOptions.cs:86` — a negative value throws `ArgumentOutOfRangeException` inside the watchdog's `Task.Delay`, `ScriptExecutionActor.cs:144`).
|
||||
|
||||
1. Failing tests (mirror the existing `Validate(...)` helper and assertion shape in the test file):
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public void ZeroTagSubscribeRetryIntervalMs_IsRejected()
|
||||
{
|
||||
var result = Validate(new SiteRuntimeOptions { TagSubscribeRetryIntervalMs = 0 });
|
||||
|
||||
Assert.True(result.Failed);
|
||||
Assert.Contains("TagSubscribeRetryIntervalMs", result.FailureMessage);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void NegativeStuckScriptGraceMs_IsRejected()
|
||||
{
|
||||
var result = Validate(new SiteRuntimeOptions { StuckScriptGraceMs = -1 });
|
||||
|
||||
Assert.True(result.Failed);
|
||||
Assert.Contains("StuckScriptGraceMs", result.FailureMessage);
|
||||
}
|
||||
```
|
||||
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~SiteRuntimeOptionsValidatorTests"` → the two new tests **FAIL** (validator succeeds on both).
|
||||
3. Implement — two `RequireThat` lines matching the file's key-naming message style:
|
||||
|
||||
```csharp
|
||||
builder.RequireThat(options.TagSubscribeRetryIntervalMs > 0,
|
||||
$"ScadaBridge:SiteRuntime:TagSubscribeRetryIntervalMs must be greater than 0 " +
|
||||
$"(was {options.TagSubscribeRetryIntervalMs}); a zero interval hot-loops the tag-subscribe " +
|
||||
"retry and a negative one throws inside the Instance Actor's retry scheduling.");
|
||||
|
||||
builder.RequireThat(options.StuckScriptGraceMs >= 0,
|
||||
$"ScadaBridge:SiteRuntime:StuckScriptGraceMs must be >= 0 " +
|
||||
$"(was {options.StuckScriptGraceMs}); a negative grace throws inside the stuck-script watchdog's delay.");
|
||||
```
|
||||
|
||||
4. Run → **PASS**, including `DefaultOptions_AreValid` (defaults 5000/30000 satisfy both rules).
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptionsValidator.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/SiteRuntimeOptionsValidatorTests.cs && git commit -m "fix(siteruntime): eagerly validate TagSubscribeRetryIntervalMs + StuckScriptGraceMs at boot (plan R2-03 T4)"`
|
||||
|
||||
### Task 5: SiteScriptCompileCache — bounded process-wide compiled-script cache (N4 groundwork)
|
||||
|
||||
**Classification:** standard (pure new class, no consumers yet)
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1, 2, 3, 4
|
||||
**Files:**
|
||||
- Create: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/SiteScriptCompileCache.cs`
|
||||
- Test: create `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/SiteScriptCompileCacheTests.cs`
|
||||
|
||||
This is the site-side adoption of the compile-cache abstraction the master tracker earmarked when PLAN-03's Task 3/4 shipped ("later refactor, not a blocker") — TemplateEngine's `ScriptCompileVerdictCache` pattern, but storing the full `ScriptCompilationResult` so both the deploy gate and the `PreStart` recompile reuse the compiled `Script<object?>` itself. Soundness: the result is a pure function of (code, globals type, compile-time-static trust policy); the stored error text is name-free (`Diagnostic.GetMessage()` / `"Compilation exception: …"` — callers prefix their own context); Roslyn `Script<T>` is immutable and thread-safe (`RunAsync` creates per-run state), so one instance may be shared across actors.
|
||||
|
||||
1. Failing tests (the type does not exist — compile failure is the RED step). The cache is process-wide static state: put this test class and Task 6's `ScriptCompilationServiceTests` additions in one xunit collection (`[Collection("SiteScriptCompileCache")]`) and call `Clear()` at the top of each test, so parallel classes cannot skew `Hits`/`Count`:
|
||||
|
||||
```csharp
|
||||
[Collection("SiteScriptCompileCache")]
|
||||
public class SiteScriptCompileCacheTests
|
||||
{
|
||||
private static ScriptCompilationResult Ok() =>
|
||||
ScriptCompilationResult.Succeeded(CSharpScript.Create<object?>("1", ScriptOptions.Default, typeof(ScriptGlobals)));
|
||||
|
||||
[Fact]
|
||||
public void GetOrAdd_SameCodeAndGlobals_ComputesOnce()
|
||||
{
|
||||
SiteScriptCompileCache.Clear();
|
||||
var factoryCalls = 0;
|
||||
ScriptCompilationResult Factory() { factoryCalls++; return Ok(); }
|
||||
|
||||
var r1 = SiteScriptCompileCache.GetOrAdd("return 1;", typeof(ScriptGlobals), Factory);
|
||||
var r2 = SiteScriptCompileCache.GetOrAdd("return 1;", typeof(ScriptGlobals), Factory);
|
||||
|
||||
Assert.Equal(1, factoryCalls);
|
||||
Assert.Same(r1, r2);
|
||||
Assert.Equal(1, SiteScriptCompileCache.Hits);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void GetOrAdd_SameCode_DifferentGlobals_AreSeparateEntries()
|
||||
{
|
||||
SiteScriptCompileCache.Clear();
|
||||
SiteScriptCompileCache.GetOrAdd("1 > 0", typeof(ScriptGlobals), Ok);
|
||||
SiteScriptCompileCache.GetOrAdd("1 > 0", typeof(TriggerExpressionGlobals), Ok);
|
||||
|
||||
Assert.Equal(2, SiteScriptCompileCache.Count);
|
||||
Assert.Equal(0, SiteScriptCompileCache.Hits); // no cross-globals hit
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Overflow_ClearsWholesale()
|
||||
{
|
||||
SiteScriptCompileCache.Clear();
|
||||
for (var i = 0; i <= SiteScriptCompileCache.MaxEntries; i++)
|
||||
SiteScriptCompileCache.GetOrAdd($"return {i};", typeof(ScriptGlobals), Ok);
|
||||
|
||||
Assert.True(SiteScriptCompileCache.Count <= SiteScriptCompileCache.MaxEntries);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~SiteScriptCompileCacheTests"` → **FAIL** (type does not exist).
|
||||
3. Implement, mirroring `ScriptCompileVerdictCache` (`src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Validation/ScriptCompileVerdictCache.cs`): `internal static class SiteScriptCompileCache` (InternalsVisibleTo already covers the test assembly) with
|
||||
- `internal const int MaxEntries = 1024;` — deliberately smaller than the verdict cache's 4096: entries pin compiled assemblies, not just verdict strings; on overflow the cache is cleared wholesale (results are recomputable, coarse reset avoids eviction bookkeeping).
|
||||
- Key = `globalsType.FullName + ":" + Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(code)))` — the globals discriminator keeps `ScriptGlobals` and `TriggerExpressionGlobals` compiles of identical source apart.
|
||||
- `public static ScriptCompilationResult GetOrAdd(string code, Type globalsType, Func<ScriptCompilationResult> factory)` over a `ConcurrentDictionary<string, ScriptCompilationResult>`, incrementing `Hits` on a hit; `Hits`/`Count`/`Clear()` exposed for tests and diagnostics — same surface as the TemplateEngine cache.
|
||||
- XML doc stating the three soundness invariants above (pure function of code+globals+static policy; name-free errors; `Script<T>` immutability) and that **failures are cached too** — a bad script body fails identically for every caller.
|
||||
4. Run → **PASS**.
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/SiteScriptCompileCache.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/SiteScriptCompileCacheTests.cs && git commit -m "feat(siteruntime): bounded process-wide compiled-script cache, keyed on code hash + globals type (plan R2-03 T5)"`
|
||||
|
||||
### Task 6: Wire the cache into ScriptCompilationService — deploy gate + PreStart share one compile per script per revision
|
||||
|
||||
**Classification:** standard (single choke point; all call sites inherit)
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** 1, 2, 3, 4 (needs 5 merged)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/ScriptCompilationService.cs` (`CompileCore`, ~lines 98–139)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs` (gate rationale comment ~lines 510–521 — comment only)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/ScriptCompilationServiceTests.cs`
|
||||
|
||||
`CompileCore` is the single funnel for every site-side compile — the `DeployCompileValidator` gate (`DeploymentManagerActor.cs:522`), the `InstanceActor.CreateChildActors` recompile (`InstanceActor.cs:1485, 1497–1498`), and `SharedScriptLibrary` all flow through it — so caching here collapses the 2×N-compiles-per-bulk-redeploy cost (N4) and shrinks P6's warm-process staggered-startup recompiles for free, with zero call-site changes.
|
||||
|
||||
1. Failing test (add to `ScriptCompilationServiceTests`, tagged `[Collection("SiteScriptCompileCache")]` per Task 5's isolation note):
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public void Compile_SameCodeTwice_SharesOneRoslynCompile()
|
||||
{
|
||||
SiteScriptCompileCache.Clear();
|
||||
var r1 = _service.Compile("deploy-gate-copy", "return 1 + 1;");
|
||||
var r2 = _service.Compile("prestart-copy", "return 1 + 1;");
|
||||
|
||||
Assert.True(r1.IsSuccess);
|
||||
Assert.Same(r1.CompiledScript, r2.CompiledScript); // one compile, shared Script<T> (N4)
|
||||
Assert.Equal(1, SiteScriptCompileCache.Hits);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Compile_ScriptAndTriggerExpression_DoNotCrossContaminate()
|
||||
{
|
||||
SiteScriptCompileCache.Clear();
|
||||
var script = _service.Compile("s", "1 > 0");
|
||||
var trigger = _service.CompileTriggerExpression("t", "1 > 0");
|
||||
|
||||
Assert.NotSame(script.CompiledScript, trigger.CompiledScript); // different globals surfaces
|
||||
}
|
||||
```
|
||||
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~Compile_SameCodeTwice_SharesOneRoslynCompile|FullyQualifiedName~Compile_ScriptAndTriggerExpression_DoNotCrossContaminate"` → **FAIL** (`Assert.Same` fails: two independent compiles today).
|
||||
3. Implement:
|
||||
- Rename the current `CompileCore` body to `private ScriptCompilationResult CompileUncached(string name, string code, Type globalsType)` (byte-identical, including the trust-model check — the verdict is deterministic on code, so it is cacheable with the compile).
|
||||
- New `CompileCore`:
|
||||
|
||||
```csharp
|
||||
private ScriptCompilationResult CompileCore(string name, string code, Type globalsType)
|
||||
{
|
||||
var result = SiteScriptCompileCache.GetOrAdd(code, globalsType,
|
||||
() => CompileUncached(name, code, globalsType));
|
||||
if (!result.IsSuccess)
|
||||
_logger.LogDebug(
|
||||
"Script {Script}: returning cached compile failure ({ErrorCount} error(s))",
|
||||
name, result.Errors.Count); // miss-path logs name the FIRST caller; keep this caller visible
|
||||
return result;
|
||||
}
|
||||
```
|
||||
|
||||
- Update the class XML doc: results (success AND failure) are memoised process-wide in `SiteScriptCompileCache`; error text is name-free by construction, so cached failures render correctly for every caller.
|
||||
- Append one sentence to the `HandleDeploy` gate comment (`DeploymentManagerActor.cs:519–521`): "Redeploys and multi-instance deploys of unchanged scripts hit the process-wide compile cache (`SiteScriptCompileCache`), so the synchronous gate recompiles only genuinely new code and the Instance Actor's PreStart reuses the gate's compile (N4)."
|
||||
4. Run the two new tests → **PASS**; then the funnel's consumer suites: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ScriptCompilationServiceTests|FullyQualifiedName~DeployCompileValidatorTests|FullyQualifiedName~SharedScriptLibraryTests|FullyQualifiedName~TrustModelSemanticTests"` → all green (these pin that caching changed no verdicts).
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/ScriptCompilationService.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/ScriptCompilationServiceTests.cs && git commit -m "perf(siteruntime): deploy gate + InstanceActor PreStart share one Roslyn compile via the compile cache (plan R2-03 T6)"`
|
||||
|
||||
### Task 7: Design-doc sync — synchronous compile gate (N5), Expression/scheduler coupling (N6), P6/N4 note refresh
|
||||
|
||||
**Classification:** trivial (docs + one stale XML doc; repo rule: docs travel with code)
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** none (write after the behavior tasks land so the docs describe reality)
|
||||
**Files:**
|
||||
- Modify: `docs/requirements/Component-SiteRuntime.md` (lines ~489, ~234–246, ~99)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Deployment/DeployCompileValidator.cs` (class XML doc, ~lines 16–18)
|
||||
|
||||
1. **N5** — `Component-SiteRuntime.md:489`: the sentence claims the gate compiles "**off-thread**"; the shipped gate is deliberately synchronous (`DeploymentManagerActor.cs:510–522`). Replace the off-thread clause so the bullet reads: "…the Deployment Manager compiles **all** of the instance's scripts, alarm on-trigger scripts, and trigger expressions **synchronously, as a pure prefix step of the deploy handler on the singleton's actor thread** — deliberately not off-thread, because piping the verdict back to self would reorder concurrent deploys against delete/disable and break the redeploy-supersede mailbox-FIFO ordering (see the in-code rationale in `DeploymentManagerActor.HandleDeploy`); unchanged script bodies hit the process-wide compile cache, so the gate recompiles only genuinely new code. Any compile failure rejects the deployment with the collected compile errors…" (keep the existing honesty tail verbatim).
|
||||
2. **N6** — `Component-SiteRuntime.md` Alarm Evaluation section (~line 236, after the trigger-type list): add one bullet: "**Expression** trigger evaluation runs on the bounded script-execution thread pool shared with script bodies (and Expression *script* triggers): scheduler saturation — e.g. stuck scripts occupying all threads — delays Expression alarm transitions site-wide until a thread frees. This is a deliberate trade-off (evaluation no longer blocks dispatcher threads; per-actor coalescing bounds queue growth) and the scheduler gauges + stuck-script watchdog make the saturated state visible."
|
||||
3. **P6/N4 note refresh** — `Component-SiteRuntime.md:99` (the existing P6 follow-up note): append "As of the round-2 hardening, a process-wide compile cache dedupes identical script bodies within a node's process lifetime (the deploy gate's compile is reused by Instance Actor start), shrinking the recompile cost; the *first* compile after process start still runs inside Instance Actor start, so the deferral stands."
|
||||
4. **N5 (code-comment drift)** — `DeployCompileValidator.cs:16–18`: the XML doc still says "it is safe to run off the singleton mailbox on a `Task.Run` thread — the deploy path never blocks on Roslyn", which describes the plan's original shape, not the shipped synchronous gate. Reword: "Because it holds no actor state and only calls the (stateless) `ScriptCompilationService`, it is pure and thread-safe; the Deployment Manager nevertheless invokes it *synchronously on the singleton's actor thread* as a pure prefix step, preserving mailbox-FIFO deploy ordering (see `HandleDeploy`). Compiles of unchanged script bodies are deduped by the process-wide compile cache."
|
||||
5. Review with `git diff docs/requirements/Component-SiteRuntime.md src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Deployment/DeployCompileValidator.cs` — verify no other "off-thread" claim about the gate remains (`grep -n "off-thread" docs/requirements/Component-SiteRuntime.md` should return only the line-99 P6 note, which refers to the *startup* compiles, not the gate).
|
||||
6. Commit: `git add docs/requirements/Component-SiteRuntime.md src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Deployment/DeployCompileValidator.cs && git commit -m "docs(siteruntime): compile gate is synchronous by design; Expression alarms share the bounded script pool (plan R2-03 T7)"`
|
||||
|
||||
---
|
||||
|
||||
## Dependencies on other plans
|
||||
|
||||
- **R2-05 (Templates/Deployment/Transport round 2)** owns the Transport options-validator knob; **R2-08** owns the AuditLog/Host validator gaps — Task 4 here covers ONLY the two SiteRuntime knobs (`TagSubscribeRetryIntervalMs`, `StuckScriptGraceMs`), per the cross-plan ruling.
|
||||
- **TemplateEngine's `ScriptCompileVerdictCache`** stays untouched: it caches *verdicts* for central validation; Task 5/6's `SiteScriptCompileCache` caches *compiled scripts* for site execution. The two are intentionally separate (different result shapes, different assemblies); no shared abstraction is introduced.
|
||||
- The site-side live-alarm-stream code (`SiteStreamManager.SubscribeSiteAlarms`) was reviewed clean in round 2 — nothing from the aggregated-live-alarm-stream plan lands here.
|
||||
|
||||
## Execution order
|
||||
|
||||
**P0 (do first):** Task 1 (N1 — the only Medium; closes the last open S1 mechanism).
|
||||
|
||||
**P1 (parallel, file-disjoint):** Tasks 2, 3 (N2 — liveness of every Expression trigger/alarm), Task 4 (N3), Task 5 (N4 groundwork).
|
||||
|
||||
**P2:** Task 6 (after 5 — consumes the cache; comment-only touch on `DeploymentManagerActor.cs`).
|
||||
|
||||
**Last:** Task 7 (docs describe the landed behavior).
|
||||
|
||||
Same-file sequencing summary: `ScriptCompilationService.cs`: 6 only. `DeploymentManagerActor.cs`: 6 only (comment). `DeployCompileValidator.cs`: 7 only (XML doc). No file is touched by two tasks.
|
||||
|
||||
After the final task: `dotnet build ZB.MOM.WW.ScadaBridge.slnx && dotnet test ZB.MOM.WW.ScadaBridge.slnx`, and for cluster-runtime verification rebuild the docker image (`bash docker/deploy.sh`).
|
||||
|
||||
---
|
||||
|
||||
## Findings Coverage
|
||||
|
||||
| Finding | Severity | Task(s) / Disposition |
|
||||
|---|---|---|
|
||||
| N1 — MxGateway: stale event loop's non-OCE fault (`RpcException(Cancelled)`/`ObjectDisposedException`) can flap the fresh connection through the generic catch; lazy `_eventLoopCts` field read in the `Task.Run` lambda | Medium | 1 (ct-guarded catch + token/client captured into locals; churn scenario regression-tested with both fault shapes) |
|
||||
| N2 — expression-eval `PipeTo` has a success mapping only; a faulted scheduler task permanently kills the trigger with `_evalInFlight` stuck | Low | 2 (ScriptActor), 3 (AlarmActor) |
|
||||
| N3 — `TagSubscribeRetryIntervalMs` / `StuckScriptGraceMs` missing from the eager options validator | Low | 4 (SiteRuntime knobs only — Transport → R2-05, AuditLog/Host → R2-08 per the cross-plan ruling) |
|
||||
| N4 — every deploy compiles each script twice (synchronous gate + `PreStart` recompile), no compile cache | Low | 5, 6 — **full adoption ships here** (one compile per script body per process; gate, PreStart, and SharedScriptLibrary all funnel through the cached `CompileCore`), so no deferred-register remainder. The gate itself stays synchronous by design (N5). |
|
||||
| N5 — spec drift (reintroduced): `Component-SiteRuntime.md:489` says the gate runs "off-thread"; the shipped gate is deliberately synchronous | Low | 7 (spec sentence + the matching stale `DeployCompileValidator` XML doc) |
|
||||
| N6 — Expression triggers/alarms share the bounded script scheduler; saturation silently stalls all Expression evaluation site-wide | Low | 7 (documented as the accepted trade-off the report judged it to be — gauges/watchdog already make it visible; no behavior change) |
|
||||
| P5 — per-attribute script read is an Ask round-trip (round-1 deferral) | Low | **Deferred (accepted, unchanged)** — spec-consistent, no profiling evidence; batch `GetAttributes` remains the profiling-gated follow-up. No task. |
|
||||
| P6 — Roslyn compiles inside `InstanceActor.PreStart` during staggered startup (round-1 deferral) | Low | **Deferred (accepted, further mitigated)** — Tasks 5/6's cache dedupes identical bodies within a warm process (the report's own observation); the first compile after process start still runs in `PreStart`, and moving it out stays out of scope. Doc note refreshed in 7. |
|
||||
| UA1 removal-reconcile / cross-node list-aggregation sub-scope (round-1 residue) | — | **Deferred** behind the documented central-persistence-of-cert-trust follow-up (`Component-SiteRuntime.md:125`). No task. |
|
||||
| S1–S10, P1–P4, C2–C6, UA1–UA7 (round-1 findings) | — | **Verified fixed** by the round-2 report's disposition table (26 fixed, evidence-read) — no action. |
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"planPath": "archreview/plans/PLAN-R2-03-site-runtime-dcl.md",
|
||||
"tasks": [
|
||||
{ "id": 1, "subject": "Task 1: MxGatewayDataConnection — stale event-loop fault on a cancelled token must not signal Disconnected (N1)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 2, "subject": "Task 2: ScriptActor — failure mapping on the expression-eval PipeTo clears _evalInFlight (N2)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 3, "subject": "Task 3: AlarmActor — failure mapping on the expression-eval PipeTo clears _evalInFlight (N2)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 4, "subject": "Task 4: Eagerly validate TagSubscribeRetryIntervalMs + StuckScriptGraceMs (N3)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 5, "subject": "Task 5: SiteScriptCompileCache — bounded process-wide compiled-script cache (N4 groundwork)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 6, "subject": "Task 6: Wire the compile cache into ScriptCompilationService — deploy gate + PreStart share one compile (N4)", "status": "pending", "blockedBy": [5] },
|
||||
{ "id": 7, "subject": "Task 7: Design-doc sync — synchronous compile gate (N5), Expression/scheduler coupling (N6), P6/N4 note refresh", "status": "pending", "blockedBy": [1, 2, 3, 4, 5, 6] }
|
||||
],
|
||||
"lastUpdated": "2026-07-13T03:23:02Z"
|
||||
}
|
||||
@@ -0,0 +1,679 @@
|
||||
# PLAN-R2-04 — Data & Audit Backbone Round-2 Fixes Implementation Plan
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
|
||||
|
||||
**Goal:** Close the seven NEW findings of the round-2 architecture review (`archreview/04-data-audit-backbone.md`, 2026-07-12): bound the KPI rollup backfill that currently re-folds the entire 90-day raw window in one tracked pass on every singleton failover (R1), make the fold fetch untracked and projected (R2), fix the ~60× rate-unit discontinuity at the raw/rollup routing boundary and the last-value-per-bucket re-fold error the aggregation catalog itself warns about (R3), close the catalog metric-literal drift hazard (R4), classify the failover fold-race failure grain (R5), give the SiteCalls terminal purge the same time-sliced batching + index support its two siblings received in round 1 (R6), and stop the site retention service from surfacing its own shutdown cancellation to the host (R7). Every accepted round-1 deferral stays deferred — coverage rows only, no code.
|
||||
|
||||
**Architecture:** All fixes stay inside the existing component boundaries. The rollup backfill/fold hardening lives in the KpiHistory recorder actor + `KpiHistoryRepository` (one additive `IKpiHistoryRepository` watermark method in Commons); the rate-presentation fix lives in the Commons `KpiSeriesBucketer` (additive optional parameter — existing call sites compile unchanged) driven by the already-shipped `KpiMetricAggregationCatalog`, wired at the `KpiHistoryQueryService` boundary; the SiteCalls purge fix rides one EF Core migration (a filtered `TerminalAtUtc` index) plus a repository-local time-sliced DELETE loop mirroring the round-1 `KpiSample` purge; the retention-service fix is a loop-boundary OCE catch. Design docs are updated in the same task as the code they describe (repo rule: doc + code + tests travel together). No new tables, no contract-breaking changes.
|
||||
|
||||
**Tech Stack:** C#/.NET 8, Akka.NET (actors/TestKit), EF Core + SQL Server (central; SQLite in-memory harness for the KpiHistory repository suite), xUnit. Build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Test per-project with targeted filters only. MSSQL-backed `SkippableFact` suites (`SiteCallAuditRepositoryTests`) need `cd infra && docker compose up -d`. **EF migrations gotcha (repo-documented):** always build first and run `dotnet ef migrations add <Name>` WITHOUT `--no-build`, or it scaffolds an empty migration off the stale DLL — delete the empty files if it happens (`migrations remove` needs a live DB).
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Untracked, projected fold fetch (R2)
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~3 min
|
||||
**Parallelizable with:** 3, 5, 6, 7, 8, 10, 11, 12 (NOT 2/9 — same repository file)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs` (`FoldHourlyRollupsAsync` fetch at :104-106; group lambdas :112-128)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/KpiHistoryRepositoryTests.cs` (extend)
|
||||
|
||||
1. Write the failing test (SQLite in-memory harness — pattern-match `NewContext()`/`Sample()` already in the file; clear the tracker after seeding so the assertion sees only the fold's own work):
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task FoldHourlyRollups_DoesNotTrack_KpiSampleEntities()
|
||||
{
|
||||
await using var ctx = NewContext();
|
||||
var repo = new KpiHistoryRepository(ctx);
|
||||
await repo.RecordSamplesAsync(new[]
|
||||
{
|
||||
Sample("NotificationOutbox", "queueDepth", "Global", null, 5, Base.AddMinutes(10)),
|
||||
Sample("NotificationOutbox", "queueDepth", "Global", null, 7, Base.AddMinutes(50)),
|
||||
});
|
||||
ctx.ChangeTracker.Clear(); // isolate the fold's tracking behavior from the seed
|
||||
|
||||
await repo.FoldHourlyRollupsAsync(Base, Base.AddHours(1));
|
||||
|
||||
// The fold READS samples and WRITES rollups; the read must not register
|
||||
// thousands of read-only KpiSample entries for DetectChanges to re-scan
|
||||
// (arch-review 04 round 2, R2).
|
||||
Assert.Empty(ctx.ChangeTracker.Entries<KpiSample>());
|
||||
}
|
||||
```
|
||||
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter FoldHourlyRollups_DoesNotTrack_KpiSampleEntities` — expect FAIL (the fetch is a tracked `ToListAsync` today).
|
||||
3. Replace the fetch at :104-106 with a projection (a `Select()` projection is never tracked, and it also drops the unneeded `Id` column, roughly halving per-row memory — which shrinks the R1 backfill's footprint even before Task 3 slices it):
|
||||
|
||||
```csharp
|
||||
// Projected, untracked fetch: the fold only reads these six fields and never
|
||||
// mutates a KpiSample. A tracked ToListAsync registered ~5k-90k read-only
|
||||
// entities in the change tracker per healthy 3h fold — all re-scanned by
|
||||
// DetectChanges on the final SaveChanges (arch-review 04 round 2, R2). A
|
||||
// projection is inherently untracked and materializes no entity at all.
|
||||
var samples = await _context.KpiSamples
|
||||
.Where(s => s.CapturedAtUtc >= from && s.CapturedAtUtc < to)
|
||||
.Select(s => new FoldSample(s.Source, s.Metric, s.Scope, s.ScopeKey, s.CapturedAtUtc, s.Value))
|
||||
.ToListAsync(cancellationToken);
|
||||
```
|
||||
|
||||
with, at the bottom of the class next to `SeriesHourKey`:
|
||||
|
||||
```csharp
|
||||
/// <summary>Narrow, untracked projection of one KpiSample row for the fold (R2).</summary>
|
||||
private readonly record struct FoldSample(
|
||||
string Source, string Metric, string Scope, string? ScopeKey, DateTime CapturedAtUtc, double Value);
|
||||
```
|
||||
|
||||
The grouping/aggregation lambdas (:112-128) compile unchanged — `FoldSample` carries the same member names.
|
||||
4. Run the new filter — PASS. Then run the whole fold suite: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter KpiHistoryRepositoryTests` — PASS (all existing fold/purge/query tests must stay green; this is a pure fetch-shape change).
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/KpiHistoryRepositoryTests.cs && git commit -m "fix(kpi): untracked projected fold fetch — fold reads no longer flood the change tracker (plan R2-04 T1)"`
|
||||
|
||||
### Task 2: Rollup watermark seam — `GetLatestRollupHourAsync` (R1, part 1)
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** 5, 8, 10, 11, 12 (NOT 1/9 — same repository file; NOT 3/4/6 — the interface addition ripples into their test fakes)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IKpiHistoryRepository.cs` (additive method)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs` (implement)
|
||||
- Modify: every test fake implementing `IKpiHistoryRepository` — locate with `grep -rln ": IKpiHistoryRepository" tests/` (expect the recorder-actor fake in `tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests/KpiHistoryRecorderActorTests.cs` and the query-service stub in `tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Services/KpiHistoryQueryServiceTests.cs`) — add a trivial `Task.FromResult<DateTime?>(null)` member so they compile
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/KpiHistoryRepositoryTests.cs` (extend)
|
||||
|
||||
1. Failing tests:
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task GetLatestRollupHour_ReturnsNull_WhenNoRollupsExist()
|
||||
{
|
||||
await using var ctx = NewContext();
|
||||
var repo = new KpiHistoryRepository(ctx);
|
||||
Assert.Null(await repo.GetLatestRollupHourAsync());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task GetLatestRollupHour_ReturnsNewestHourStart_AcrossAllSeries()
|
||||
{
|
||||
await using var ctx = NewContext();
|
||||
var repo = new KpiHistoryRepository(ctx);
|
||||
await repo.RecordSamplesAsync(new[]
|
||||
{
|
||||
Sample("NotificationOutbox", "queueDepth", "Global", null, 5, Base.AddMinutes(10)),
|
||||
Sample("SiteCallAudit", "buffered", "Global", null, 2, Base.AddHours(3).AddMinutes(10)),
|
||||
});
|
||||
await repo.FoldHourlyRollupsAsync(Base, Base.AddHours(4));
|
||||
|
||||
Assert.Equal(Base.AddHours(3), await repo.GetLatestRollupHourAsync());
|
||||
}
|
||||
```
|
||||
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter GetLatestRollupHour` — expect FAIL (method does not exist / does not compile — comment the tests out to prove the baseline if needed, standard TDD-on-new-member flow).
|
||||
3. Add to `IKpiHistoryRepository` (after `GetHourlySeriesAsync`, ~:108):
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Returns the newest <c>KpiRollupHourly.HourStartUtc</c> across all series, or
|
||||
/// <c>null</c> when no rollups exist. The recorder's one-shot backfill consults
|
||||
/// this watermark to skip (or shrink to) the un-rolled tail instead of re-folding
|
||||
/// the whole raw-retention window on every failover (arch-review 04 round 2, R1).
|
||||
/// </summary>
|
||||
Task<DateTime?> GetLatestRollupHourAsync(CancellationToken cancellationToken = default);
|
||||
```
|
||||
|
||||
Implement in `KpiHistoryRepository`: `return await _context.KpiRollupHourly.MaxAsync(r => (DateTime?)r.HourStartUtc, cancellationToken);` — a single MAX over the unique `IX_KpiRollupHourly_Series` population (rollup table is small: one row per series-hour). Add the null-returning member to each located test fake.
|
||||
4. Run the filter — PASS. Build the solution (`dotnet build ZB.MOM.WW.ScadaBridge.slnx`) to prove no other implementor was missed.
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IKpiHistoryRepository.cs src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs tests && git commit -m "feat(kpi): GetLatestRollupHourAsync watermark seam for the backfill fast-path (plan R2-04 T2)"`
|
||||
|
||||
### Task 3: Slice the rollup backfill into bounded day windows (R1, part 2)
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 5, 6, 7, 8, 10, 11, 12 (NOT 4 — same actor file; requires Task 2's fake update)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.KpiHistory/KpiHistoryRecorderActor.cs` (`HandleBackfillTick` :494-526, `RunBackfillPass` :536-558, the `BackfillComplete` receive :163-167)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests/KpiHistoryRecorderActorTests.cs` (extend + update the two existing backfill tests)
|
||||
|
||||
1. Failing test (reuse the gated-fold fake-repository rig the existing `BackfillTick_FoldsFullRetentionWindow_Once` / `BackfillInFlight_SkipsPeriodicRollupTick` tests drive by hand):
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public void Backfill_SlicesRetentionWindow_IntoBoundedDayFolds_OldestFirst()
|
||||
{
|
||||
// RetentionDays = 3 → expect 3+ fold calls, each window ≤ 24 h, contiguous,
|
||||
// oldest-first, whose union is exactly [TruncateToHour(now) − 3 d, TruncateToHour(now)).
|
||||
// Today: FAIL — a single fold call spans the whole 3-day window.
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void PeriodicRollupTick_Interleaves_BetweenBackfillSlices()
|
||||
{
|
||||
// Gate slice 1 open; release it; before dequeuing slice 2 send a RollupTick —
|
||||
// the periodic fold must claim the fold path (the backfill defers via its
|
||||
// re-arm branch) and ALL backfill slices must still complete afterwards.
|
||||
// Today: FAIL — _backfillInFlight blocks every periodic fold for the whole pass.
|
||||
}
|
||||
```
|
||||
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests --filter "Backfill_SlicesRetentionWindow|PeriodicRollupTick_Interleaves"` — expect FAIL.
|
||||
3. Implementation — convert the single full-window pass into a slice queue processed one bounded window per fold, releasing the fold path between slices:
|
||||
- New constant + state:
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Width of one backfill fold slice (24 h). The repository fold materializes its whole
|
||||
/// window in memory, so the one-shot backfill must never hand it the full raw-retention
|
||||
/// window (90 d ≈ tens of millions of rows at fleet volume — arch-review 04 round 2, R1).
|
||||
/// One day per fold bounds each pass to roughly what a steady-state day writes, and the
|
||||
/// guard is lowered between slices so periodic folds interleave instead of stalling
|
||||
/// behind the historical pass.
|
||||
/// </summary>
|
||||
private static readonly TimeSpan BackfillSliceWidth = TimeSpan.FromHours(24);
|
||||
|
||||
private readonly Queue<(DateTime FromHourUtc, DateTime ToHourUtc)> _backfillSlices = new();
|
||||
```
|
||||
|
||||
- `HandleBackfillTick`: keep the `_backfillDone || _backfillInFlight` and `_rollupInFlight`-defer branches verbatim. When `_backfillSlices` is empty, build the plan — enumerate `[toHourUtc − RetentionDays, toHourUtc)` into ≤24h windows oldest-first and enqueue them (log the one "backfill starting" line with the slice count); when it is non-empty, dequeue ONE slice, raise `_backfillInFlight`, and run `RunBackfillPass(slice.FromHourUtc, slice.ToHourUtc, ct)` piping `BackfillSliceComplete.Instance` on both success and (logged) failure.
|
||||
- Replace the `BackfillComplete` receive with:
|
||||
|
||||
```csharp
|
||||
Receive<BackfillSliceComplete>(_ =>
|
||||
{
|
||||
_backfillInFlight = false; // release the fold path BETWEEN slices (R1)
|
||||
if (_backfillSlices.Count == 0)
|
||||
{
|
||||
_backfillDone = true; // latch: at most once per actor lifetime (unchanged semantics)
|
||||
_logger.LogInformation("KPI rollup backfill completed — all slices folded.");
|
||||
return;
|
||||
}
|
||||
|
||||
// Next slice via the timer, not inline: a RollupTick already queued in the mailbox
|
||||
// is processed first, so periodic folds interleave; the existing _rollupInFlight
|
||||
// deferral branch in HandleBackfillTick then re-arms the backfill behind it.
|
||||
Timers.StartSingleTimer(BackfillTimerKey, BackfillTick.Instance, TimeSpan.Zero);
|
||||
});
|
||||
```
|
||||
|
||||
Rename the `BackfillComplete` message class to `BackfillSliceComplete` (internal — used only by this actor and its tests). Move the "backfill completed" wall-time log from `RunBackfillPass` (which now logs per-slice at Debug) to the queue-drained branch. Ordering note in the class doc: periodic folds and backfill slices strictly alternate through the single `_rollupInFlight`/`_backfillInFlight` gate pair — the two idempotent upserts still never race on overlapping rows.
|
||||
- Update `BackfillTick_FoldsFullRetentionWindow_Once` (assert the union of slice windows covers the retention window and a post-completion `BackfillTick` is a no-op) and `BackfillInFlight_SkipsPeriodicRollupTick` (unchanged semantics — a RollupTick arriving while a *slice* is in flight is still skipped).
|
||||
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests --filter KpiHistoryRecorderActorTests` — PASS (all pre-existing sample/purge/rollup tests must stay green).
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.KpiHistory/KpiHistoryRecorderActor.cs tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests/KpiHistoryRecorderActorTests.cs && git commit -m "fix(kpi): slice the rollup backfill into bounded 24h folds that yield to periodic folds (plan R2-04 T3)"`
|
||||
|
||||
### Task 4: Backfill failover fast-path via the rollup watermark + doc correction (R1, part 3)
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 5, 6, 8, 10, 11, 12 (NOT 3 — same actor file, do 3 first; NOT 7/9 — same design-doc file)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.KpiHistory/KpiHistoryRecorderActor.cs` (plan branch of `HandleBackfillTick` from Task 3)
|
||||
- Modify: `docs/requirements/Component-KpiHistory.md` (:93, the "One-shot backfill" paragraph)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests/KpiHistoryRecorderActorTests.cs` (extend)
|
||||
|
||||
1. Failing tests (the fake repository's `GetLatestRollupHourAsync` — added in Task 2 — gains a settable return value):
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public void Backfill_Skips_WhenRollupsAlreadyCurrent()
|
||||
{
|
||||
// Fake watermark = TruncateToHour(now) − 1 h (i.e. within RollupLookbackHours of now —
|
||||
// the failover case, which is the common one): expect ZERO fold calls, the done-latch
|
||||
// set, and an Information log; the periodic fold alone covers the lookback tail.
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Backfill_ShrinksToWatermark_InsteadOfRetentionFloor()
|
||||
{
|
||||
// RetentionDays = 90, fake watermark = 2 days ago: expect the first slice to start at
|
||||
// the WATERMARK HOUR (inclusive — re-folding the newest already-folded hour is a cheap
|
||||
// idempotent safety margin), not 90 days ago, and ≤ 3 slices total.
|
||||
}
|
||||
```
|
||||
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests --filter "Backfill_Skips_WhenRollupsAlreadyCurrent|Backfill_ShrinksToWatermark"` — expect FAIL (the plan branch always enumerates from the retention floor).
|
||||
3. Implementation — make the plan branch asynchronous: when `_backfillSlices` is empty, raise `_backfillInFlight` and run a plan pass that resolves `IKpiHistoryRepository` in a fresh scope and returns `GetLatestRollupHourAsync()` (never-faulting wrapper like the other passes), piping `BackfillPlan(DateTime? Watermark)` back to Self. In the `Receive<BackfillPlan>` handler (actor thread):
|
||||
|
||||
```csharp
|
||||
var toHourUtc = TruncateToHour(DateTime.UtcNow);
|
||||
var retentionFloor = toHourUtc - TimeSpan.FromDays(_options.RetentionDays);
|
||||
// Shrink to the un-rolled tail; re-fold the watermark hour itself (idempotent) as a
|
||||
// safety margin. A null watermark (fresh install) keeps the full retention window.
|
||||
var floor = plan.Watermark is { } w && w > retentionFloor ? w : retentionFloor;
|
||||
|
||||
if (floor >= toHourUtc - TimeSpan.FromHours(_options.RollupLookbackHours))
|
||||
{
|
||||
// Failover fast-path: rollups are already current — the periodic fold's lookback
|
||||
// window covers everything from the watermark forward, so the full historical
|
||||
// pass is skipped entirely (arch-review 04 round 2, R1).
|
||||
_backfillInFlight = false;
|
||||
_backfillDone = true;
|
||||
_logger.LogInformation(
|
||||
"KPI rollup backfill skipped — rollups current through {Watermark:o}.", plan.Watermark);
|
||||
return;
|
||||
}
|
||||
|
||||
// enqueue ≤24h slices over [floor, toHourUtc) oldest-first (Task 3's builder),
|
||||
// lower _backfillInFlight, then self-arm BackfillTick to dequeue the first slice.
|
||||
```
|
||||
|
||||
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests --filter KpiHistoryRecorderActorTests` — PASS.
|
||||
5. Doc: rewrite `Component-KpiHistory.md:93` — the backfill folds the window in **≤24h slices oldest-first**, yielding to periodic folds between slices; on a failover restart it consults the `GetLatestRollupHourAsync` watermark and **skips** (rollups current) or **shrinks to the un-rolled tail**. Drop the "cheap and safe to re-run" claim — replace with "safe (idempotent) to re-run; the watermark fast-path makes the failover re-run cheap as well" (the review's exact complaint: the "safe" was true, the "cheap" was not).
|
||||
6. Commit: `git add src/ZB.MOM.WW.ScadaBridge.KpiHistory/KpiHistoryRecorderActor.cs docs/requirements/Component-KpiHistory.md tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests/KpiHistoryRecorderActorTests.cs && git commit -m "fix(kpi): backfill skips or shrinks to the rollup watermark on failover restarts (plan R2-04 T4)"`
|
||||
|
||||
### Task 5: Per-metric reduction in the bucketer — sum-per-bucket for Rate series (R3, part 1)
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1, 2, 3, 4, 8, 10, 11, 12
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Types/Kpi/KpiSeriesBucketer.cs`
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Kpi/KpiSeriesBucketerTests.cs` (extend)
|
||||
|
||||
1. Failing tests:
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public void Bucket_RateSeries_SumsPerBucket_InsteadOfLastValue()
|
||||
{
|
||||
// 6 points of value 10 across a window bucketed to 2 → each bucket = 30 (sum),
|
||||
// not 10 (last value). Today: FAIL (last-value keeps 1 of 3 deltas per bucket —
|
||||
// the exact "keep one delta, discard the others" fold error the catalog warns
|
||||
// about, re-introduced one layer up: arch-review 04 round 2, R3).
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Bucket_RateSeries_ShortSeries_StillSumsClusteredPoints()
|
||||
{
|
||||
// 3 points (< maxPoints) with two sharing a bucket → the shared bucket is their SUM.
|
||||
// Rate series skip the raw.Count <= maxPoints early return so totals stay truthful.
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Bucket_RateSeries_PreservesSeriesTotal()
|
||||
{
|
||||
// Strong invariant: sum(output values) == sum(in-window input values) for Rate.
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Bucket_DefaultAggregation_IsGauge_AndBehaviorUnchanged()
|
||||
{
|
||||
// Calling the existing 4-arg shape produces byte-identical output to before.
|
||||
}
|
||||
```
|
||||
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests --filter KpiSeriesBucketerTests` — expect FAIL.
|
||||
3. Implementation — additive optional parameter (every existing call site compiles unchanged):
|
||||
|
||||
```csharp
|
||||
public static IReadOnlyList<KpiSeriesPoint> Bucket(
|
||||
IReadOnlyList<KpiSeriesPoint> raw,
|
||||
DateTime fromUtc,
|
||||
DateTime toUtc,
|
||||
int maxPoints,
|
||||
KpiRollupAggregation aggregation = KpiRollupAggregation.Gauge)
|
||||
```
|
||||
|
||||
- Keep the argument guards and the null/empty return unchanged. Restrict the `raw.Count <= maxPoints` early return to **Gauge only**: for Rate, two deltas landing in one bucket must sum even in a short series (for well-spaced short series the per-point buckets make sum == identity, so output ≈ input anyway).
|
||||
- In the loop, branch per aggregation: Gauge keeps the existing last-value candidate logic verbatim; Rate accumulates `sums[bucketIndex] += point.Value` (plus the same `occupied` flag), emitting `new KpiSeriesPoint(bucketStart, sums[i])` in the collection pass.
|
||||
- Update the class/method XML doc: "last-value-per-bucket / gauge semantics" becomes per-metric — Gauge = last value in the bucket (unchanged default), Rate = **sum-per-bucket** (each raw point of a Rate series is a per-interval delta, so the bucket's true total is the sum of its deltas; both the raw-minute and hourly-rollup paths then chart the same unit, "events per chart bucket", eliminating the ~60× jump at the routing boundary — R3).
|
||||
4. Run the filter — PASS. Also `dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests --filter "FullyQualifiedName~Kpi"` — PASS (catalog + point tests untouched).
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Commons/Types/Kpi/KpiSeriesBucketer.cs tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Kpi/KpiSeriesBucketerTests.cs && git commit -m "fix(kpi): per-metric bucket reduction — Rate series sum per bucket instead of keeping one delta (plan R2-04 T5)"`
|
||||
|
||||
### Task 6: Catalog-driven aggregation at the query-service boundary (R3, part 2)
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** 1, 3, 4, 8, 10, 11, 12 (requires Task 5)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/KpiHistoryQueryService.cs` (`GetSeriesAsync` :73-94, `FetchSeriesAsync` doc :96-106)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Services/KpiHistoryQueryServiceTests.cs` (extend)
|
||||
|
||||
1. Failing tests (stub-repository rig already in the file):
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task GetSeriesAsync_RateMetric_SumsPerBucket_OnRollupPath()
|
||||
{
|
||||
// (KpiSources.SiteCallAudit, KpiMetrics.SiteCallAudit.FailedLastInterval) is a
|
||||
// catalog Rate pair. Stub GetHourlySeriesAsync with 400 hourly points of value 1
|
||||
// over a >168h window, maxPoints 200 → every output value is 2.0 (sum of the two
|
||||
// hourly sums per bucket), not 1.0 (last value). Today: FAIL.
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task GetSeriesAsync_RateMetric_PreservesTotal_AcrossRoutingBoundary()
|
||||
{
|
||||
// Same total event count served as per-minute deltas (raw path, window == threshold)
|
||||
// and as hourly sums (rollup path, window just over threshold): the summed chart
|
||||
// total must be equal on both paths — the boundary is now visually seamless (R3).
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task GetSeriesAsync_GaugeMetric_KeepsLastValuePerBucket()
|
||||
{
|
||||
// queueDepth (uncatalogued → Gauge) keeps the existing last-value output unchanged.
|
||||
}
|
||||
```
|
||||
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests --filter KpiHistoryQueryServiceTests` — expect FAIL.
|
||||
3. Implementation in `GetSeriesAsync` (both ctor paths):
|
||||
|
||||
```csharp
|
||||
// Per-metric reduction intent: the same catalog that drives the hourly fold drives
|
||||
// the chart-time bucket reduction, so a Rate series is summed per bucket on BOTH the
|
||||
// raw-minute and hourly-rollup routes — one unit ("events per chart bucket") on both
|
||||
// sides of the RollupThresholdHours routing boundary (arch-review 04 round 2, R3).
|
||||
var aggregation = KpiMetricAggregationCatalog.Resolve(source, metric);
|
||||
...
|
||||
return KpiSeriesBucketer.Bucket(injectedSeries, fromUtc, toUtc, effectiveMax, aggregation);
|
||||
...
|
||||
return KpiSeriesBucketer.Bucket(series, fromUtc, toUtc, effectiveMax, aggregation);
|
||||
```
|
||||
|
||||
Extend the `FetchSeriesAsync` doc comment: the routing boundary no longer changes a Rate chart's magnitude — both paths reduce to per-bucket sums (a residual ≤~1.2× native-resolution difference exists only for an unbucketed just-over-threshold rollup series vs. an at-threshold bucketed raw series; documented in Task 7).
|
||||
4. Run the filter — PASS (the pre-existing routing/forwarding tests must stay green — Gauge default preserves their expectations).
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/KpiHistoryQueryService.cs tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Services/KpiHistoryQueryServiceTests.cs && git commit -m "fix(kpi): query service applies catalog aggregation at the bucketer — rate charts keep one unit across the raw/rollup boundary (plan R2-04 T6)"`
|
||||
|
||||
### Task 7: Truthful trend presentation — chart doc + Component-KpiHistory.md (R3, part 3)
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~3 min
|
||||
**Parallelizable with:** 1, 2, 3, 8, 10, 11, 12 (requires Task 6; NOT 4/9 — same design-doc file)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Shared/KpiTrendChart.razor.cs` (XML docs on `Points` :~55 and `Unit` :~62 — doc-only, no markup/behavior change; the component stays custom-SVG, no third-party libs)
|
||||
- Modify: `docs/requirements/Component-KpiHistory.md` (:151 "Bucketed query", the "Raw-vs-rollup range routing" section :~153-155, and one line under ":157 Aggregation intent")
|
||||
- Modify: trend-panel `Unit` strings only where they now misstate the semantics — locate with `grep -rn "Unit=" src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages` and fix any Rate-metric panel whose unit text claims a per-minute/per-interval reading (e.g. the `deliveredLastInterval` / `failedLastInterval` panels) to say per-bucket totals (e.g. `Unit="delivered/bucket"`); leave Gauge panels untouched
|
||||
|
||||
1. `KpiTrendChart.razor.cs` doc: on `Points`, note that Rate-classified series (per `KpiMetricAggregationCatalog`) now carry **per-bucket totals** (sum reduction), while Gauge series carry last-value-per-bucket readings; on `Unit`, note the supplied text must state the per-bucket semantics for Rate metrics.
|
||||
2. `Component-KpiHistory.md:151`: replace "returns the **last value per bucket**" with the per-metric reduction — `KpiSeriesBucketer.Bucket(series, fromUtc, toUtc, maxPoints, aggregation)` keeps the last value for Gauge series and **sums per bucket** for Rate series, driven by the same `KpiMetricAggregationCatalog` the fold consults (the previous last-value-on-rollups behavior kept 1 of ~11 hourly sums per 90d bucket — the fold error the catalog's own doc warns about, fixed per arch-review 04 round 2, R3). In the range-routing section add: "Rate charts keep one unit (events per chart bucket) on both sides of the boundary; widening 7d→30d no longer changes the magnitude ~60×. Residual: an unbucketed just-over-threshold rollup series plots native per-hour sums vs. the at-threshold raw path's ~50-minute bucket sums (≤~1.2×) — accepted and documented, not hidden." Under Aggregation intent, note the catalog now has two consumers: the hourly fold and the chart-time bucket reduction.
|
||||
3. Verify no stale claim remains: `grep -n "last value per bucket" docs/requirements/Component-KpiHistory.md` returns nothing.
|
||||
4. Run the chart component suite to prove the doc-only touch broke nothing: `dotnet test tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests --filter KpiTrendChartTests` — PASS.
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.CentralUI docs/requirements/Component-KpiHistory.md && git commit -m "docs(kpi): trend charts state per-bucket-total semantics for rate metrics (plan R2-04 T7)"`
|
||||
|
||||
### Task 8: Close the catalog metric-literal drift hazard (R4)
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** 1, 2, 3, 4, 5, 10, 11, 12
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Types/Kpi/KpiMetrics.cs` (add three constants + amend the class remark)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Types/Kpi/KpiMetricAggregation.cs` (:60-64 private literals; :98-107 `RatePairs`)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/Kpi/SiteCallAuditKpiSampleSource.cs` (:45)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/Kpi/SiteHealthKpiSampleSource.cs` (:43, :51)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Kpi/KpiMetricAggregationTests.cs` (extend)
|
||||
|
||||
1. Failing test (does not compile until the constants exist — the TDD gate for symbol promotion):
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public void PromotedMetricConstants_LockHistoricalPersistedValues()
|
||||
{
|
||||
// Persisted data contract — these are symbol promotions, NOT renames.
|
||||
Assert.Equal("deliveredLastInterval", KpiMetrics.SiteCallAudit.DeliveredLastInterval);
|
||||
Assert.Equal("alarmEvalErrors", KpiMetrics.SiteHealth.AlarmEvalErrors);
|
||||
Assert.Equal("eventLogWriteFailures", KpiMetrics.SiteHealth.EventLogWriteFailures);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(KpiSources.SiteCallAudit, KpiMetrics.SiteCallAudit.DeliveredLastInterval)]
|
||||
[InlineData(KpiSources.SiteHealth, KpiMetrics.SiteHealth.AlarmEvalErrors)]
|
||||
[InlineData(KpiSources.SiteHealth, KpiMetrics.SiteHealth.EventLogWriteFailures)]
|
||||
public void Resolve_PromotedRatePairs_StillClassifyAsRate(string source, string metric)
|
||||
=> Assert.Equal(KpiRollupAggregation.Rate, KpiMetricAggregationCatalog.Resolve(source, metric));
|
||||
```
|
||||
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests --filter KpiMetricAggregationTests` — expect FAIL (compile error: constants absent).
|
||||
3. Implementation — compile-time coupling in the only dependency-legal direction (Commons cannot reference the source projects, so the shared constant lives in `KpiMetrics` and both the catalog and the emitting source consume it):
|
||||
- `KpiMetrics.SiteCallAudit`: add `public const string DeliveredLastInterval = "deliveredLastInterval";`; `KpiMetrics.SiteHealth`: add `AlarmEvalErrors = "alarmEvalErrors"` and `EventLogWriteFailures = "eventLogWriteFailures"` (EXACT existing string values — persisted, renames forbidden).
|
||||
- Amend the `KpiMetrics` class remark (:20-27): the catalog now also carries metrics the **rollup aggregation catalog** classifies, not only charted ones — the C4/R4 lesson is that any metric named in two places needs one shared symbol.
|
||||
- `KpiMetricAggregationCatalog`: delete the three private literals (:62-64) and reference the constants in `RatePairs`; update the "otherwise the source's own private literal" remark (:46-48) — every catalogued name is now a shared `KpiMetrics` constant, so an emitter rename is a compile error, not a silent Gauge downgrade.
|
||||
- The two sources keep their private-const style with the value redefined in terms of the shared symbol (tiny diff, zero call-site churn): `private const string MetricDeliveredLastInterval = KpiMetrics.SiteCallAudit.DeliveredLastInterval;` (SiteCallAudit :45) and likewise for the two SiteHealth consts (:43, :51).
|
||||
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests --filter KpiMetricAggregationTests && dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests --filter SiteCallAuditKpiSampleSource && dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests --filter SiteHealthKpiSampleSource` — PASS (emitted strings unchanged).
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Commons src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/Kpi/SiteCallAuditKpiSampleSource.cs src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/Kpi/SiteHealthKpiSampleSource.cs tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Kpi/KpiMetricAggregationTests.cs && git commit -m "fix(kpi): promote catalog metric literals to shared KpiMetrics constants — drift becomes a compile error (plan R2-04 T8)"`
|
||||
|
||||
### Task 9: Classify the failover fold-race failure grain (R5)
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** 3, 5, 6, 8, 10, 11, 12 (NOT 1/2 — same repository file, do them first; NOT 4/7 — same design-doc file)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs` (ctor + the fold's `SaveChangesAsync` at :167)
|
||||
- Modify: `docs/requirements/Component-KpiHistory.md` (one sentence in the "Hourly rollup tick" paragraph, :~91)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/KpiHistoryRepositoryTests.cs` (extend)
|
||||
|
||||
1. Failing test — force a REAL unique violation deterministically with a `SaveChangesInterceptor` that, on the fold's first save, inserts the conflicting `(series, hour)` rollup row through a separate command on the same in-memory SQLite connection (simulating the old singleton incarnation's in-flight fold winning the race):
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task Fold_LosingFailoverUpsertRace_DoesNotThrow_AndNextFoldSelfHeals()
|
||||
{
|
||||
// ctx built with .AddInterceptors(new InsertConflictingRollupOnFirstSave(connection))
|
||||
// seed one sample in a complete hour; FoldHourlyRollupsAsync must NOT propagate
|
||||
// DbUpdateException (today: FAIL — the whole pass faults);
|
||||
// a second FoldHourlyRollupsAsync over the same window must land the row (self-heal).
|
||||
}
|
||||
```
|
||||
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter Fold_LosingFailoverUpsertRace` — expect FAIL.
|
||||
3. Implementation:
|
||||
- Ctor gains `ILogger<KpiHistoryRepository>? logger = null` defaulting to `NullLogger` (mirrors `SiteCallAuditRepository` — MS.DI resolves `ILogger<>` automatically, no registration churn).
|
||||
- Wrap the fold's final `SaveChangesAsync`:
|
||||
|
||||
```csharp
|
||||
try
|
||||
{
|
||||
await _context.SaveChangesAsync(cancellationToken);
|
||||
}
|
||||
catch (DbUpdateException ex)
|
||||
{
|
||||
// Failover-overlap race: the old singleton incarnation's in-flight fold and the
|
||||
// new node's first fold can both Add the same (series, hour) row; the unique
|
||||
// IX_KpiRollupHourly_Series then faults the loser's ENTIRE SaveChanges — the
|
||||
// failure grain is the PASS, not the row (arch-review 04 round 2, R5). The fold
|
||||
// only ever writes KpiRollupHourly rows, so any save fault here is a lost upsert
|
||||
// race or a transient the next idempotent re-fold repairs identically; classify
|
||||
// at Information instead of surfacing a scary error for a self-healing no-op.
|
||||
_logger.LogInformation(ex,
|
||||
"KPI rollup fold lost a failover-overlap upsert race; pass discarded, next fold self-heals.");
|
||||
}
|
||||
```
|
||||
|
||||
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter KpiHistoryRepositoryTests` — PASS.
|
||||
5. Doc: add to the rollup-tick paragraph of `Component-KpiHistory.md`: "During a failover overlap the loser of the `(series, hour)` upsert race discards its whole fold pass (single `SaveChanges`) — logged at Information, repaired by the next idempotent fold; when reading fold logs after a failover, the failure grain is the pass, not the row."
|
||||
6. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs docs/requirements/Component-KpiHistory.md tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/KpiHistoryRepositoryTests.cs && git commit -m "fix(kpi): classify the failover fold-race pass loss as a self-healing Information event (plan R2-04 T9)"`
|
||||
|
||||
### Task 10: SiteCalls filtered terminal index (migration) (R6, part 1)
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** 1, 2, 3, 4, 5, 6, 7, 8, 9, 12 (the plan's only migration — but serialize against any OTHER plan adding ConfigurationDatabase migrations: model-snapshot ordering)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/SiteCallEntityTypeConfiguration.cs` (after the `IX_SiteCalls_NonTerminal` block ending :~99)
|
||||
- Create: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/<timestamp>_AddSiteCallsTerminalIndex.cs` (scaffolded)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/SchemaConfigurationTests.cs` (extend)
|
||||
|
||||
1. Failing model test (SchemaConfigurationTests already asserts index shapes):
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public void SiteCalls_Has_Filtered_Terminal_Index()
|
||||
{
|
||||
var entity = Model.FindEntityType(typeof(SiteCall))!;
|
||||
var index = entity.GetIndexes().Single(i => i.GetDatabaseName() == "IX_SiteCalls_Terminal");
|
||||
Assert.Equal(new[] { nameof(SiteCall.TerminalAtUtc) }, index.Properties.Select(p => p.Name).ToArray());
|
||||
Assert.Equal("[TerminalAtUtc] IS NOT NULL", index.GetFilter());
|
||||
}
|
||||
```
|
||||
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter SiteCalls_Has_Filtered_Terminal_Index` — expect FAIL.
|
||||
3. Add to `SiteCallEntityTypeConfiguration.Configure`:
|
||||
|
||||
```csharp
|
||||
// Terminal backs the daily retention purge's predicate (TerminalAtUtc IS NOT NULL
|
||||
// AND TerminalAtUtc < cutoff) and Task 11's MIN() slice anchor: filtered to the
|
||||
// terminal population so the purge seeks the expired tail instead of full-scanning
|
||||
// the 365-day table (arch-review 04 round 2, R6). Complements IX_SiteCalls_NonTerminal,
|
||||
// which is filtered to IS NULL and unusable for this predicate.
|
||||
builder.HasIndex(s => s.TerminalAtUtc)
|
||||
.HasDatabaseName("IX_SiteCalls_Terminal")
|
||||
.HasFilter("[TerminalAtUtc] IS NOT NULL");
|
||||
```
|
||||
|
||||
4. Scaffold the migration — **repo gotcha: BUILD FIRST, never `dotnet ef migrations add --no-build`** (it scaffolds an EMPTY migration off the stale previously-built DLL; if that happens, delete the empty `Migrations/` files — `migrations remove` needs a live DB):
|
||||
|
||||
```bash
|
||||
dotnet build ZB.MOM.WW.ScadaBridge.slnx
|
||||
cd src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase
|
||||
dotnet ef migrations add AddSiteCallsTerminalIndex
|
||||
```
|
||||
|
||||
Verify the scaffold contains the `CreateIndex` with the `IS NOT NULL` filter — if it is empty, the gotcha bit; delete and redo.
|
||||
5. Run the model test — PASS. With infra up (`cd infra && docker compose up -d`), run the MSSQL-backed suite: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter SiteCallAuditRepositoryTests` — PASS. Production script (repo convention — manual SQL for production): `dotnet ef migrations script AddKpiRollupHourlyTable AddSiteCallsTerminalIndex --idempotent --output ../../docs/plans/sql/AddSiteCallsTerminalIndex.sql` (verify the from-migration is still the latest applied; review the output).
|
||||
6. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/SchemaConfigurationTests.cs docs/plans/sql && git commit -m "perf(sitecallaudit): filtered IX_SiteCalls_Terminal index backs the retention purge predicate (plan R2-04 T10)"`
|
||||
|
||||
### Task 11: Time-sliced SiteCalls terminal purge (R6, part 2)
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** 1, 2, 3, 4, 5, 6, 7, 8, 9, 12 (requires Task 10 — the MIN() anchor and DELETE predicate seek its index)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SiteCallAuditRepository.cs` (`PurgeTerminalAsync` :247-252)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/SiteCallAuditRepositoryTests.cs` (extend; MSSQL `SkippableFact` — infra up)
|
||||
|
||||
1. Failing test (pattern-match the fixture usage of the existing `SkippableFact` tests in the file; assert slicing via an EF `DbCommandInterceptor` capturing DELETE statements, or SQL capture matching how sibling tests inspect SQL):
|
||||
|
||||
```csharp
|
||||
[SkippableFact]
|
||||
public async Task PurgeTerminal_SlicesMultiDayBacklog_IntoBoundedDeletes()
|
||||
{
|
||||
// Seed terminal rows spread across 3 distinct days beyond the cutoff, one terminal
|
||||
// row inside the window, and one OLD non-terminal row.
|
||||
// Purge → returns 3-day total; the fresh terminal row and the old NON-terminal row
|
||||
// survive (non-terminal rows are NEVER purged on age — sites remain source of truth);
|
||||
// captured SQL shows MORE THAN ONE DELETE statement for the multi-day backlog.
|
||||
// Today: FAIL — a single unbatched DELETE is issued.
|
||||
}
|
||||
```
|
||||
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter PurgeTerminal_SlicesMultiDayBacklog` — expect FAIL.
|
||||
3. Implement time-sliced batching mirroring `KpiHistoryRepository.PurgeOlderThanAsync` (the round-1 Task 19 shape), keeping the raw-T-SQL MSSQL-only style this repository already uses:
|
||||
|
||||
```csharp
|
||||
public async Task<int> PurgeTerminalAsync(DateTime olderThanUtc, CancellationToken ct = default)
|
||||
{
|
||||
// Time-sliced batches (arch-review 04 round 2, R6 — the one maintenance DELETE
|
||||
// that missed round 1's batching pass): each DELETE covers at most one DAY of
|
||||
// terminal rows, capping the lock/log footprint per statement. Steady state
|
||||
// (daily purge, 365-day retention) is a single slice; only catch-up after an
|
||||
// outage runs several. One-day (not one-hour) slices are proportionate to
|
||||
// SiteCalls volume, which is far below KpiSample's. The MIN() anchor and the
|
||||
// DELETE predicate both seek IX_SiteCalls_Terminal (filtered IS NOT NULL).
|
||||
var total = 0;
|
||||
var floor = await _context.SiteCalls
|
||||
.Where(s => s.TerminalAtUtc != null && s.TerminalAtUtc < olderThanUtc)
|
||||
.MinAsync(s => s.TerminalAtUtc, ct);
|
||||
while (floor is not null && floor < olderThanUtc)
|
||||
{
|
||||
var ceiling = floor.Value.AddDays(1) < olderThanUtc ? floor.Value.AddDays(1) : olderThanUtc;
|
||||
total += await _context.Database.ExecuteSqlInterpolatedAsync(
|
||||
$"DELETE FROM dbo.SiteCalls WHERE TerminalAtUtc IS NOT NULL AND TerminalAtUtc < {ceiling};",
|
||||
ct);
|
||||
floor = await _context.SiteCalls
|
||||
.Where(s => s.TerminalAtUtc != null && s.TerminalAtUtc < olderThanUtc)
|
||||
.MinAsync(s => s.TerminalAtUtc, ct);
|
||||
}
|
||||
|
||||
return total;
|
||||
}
|
||||
```
|
||||
|
||||
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter SiteCallAuditRepositoryTests` — PASS (upsert/query/KPI tests untouched). Then the purge-actor contract: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests --filter SiteCallAuditPurgeTests` — PASS (return-total contract unchanged).
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SiteCallAuditRepository.cs tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/SiteCallAuditRepositoryTests.cs && git commit -m "fix(sitecallaudit): time-sliced terminal purge replaces the single year-scale DELETE (plan R2-04 T11)"`
|
||||
|
||||
### Task 12: Retention-service shutdown no longer surfaces cancellation (R7)
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~3 min
|
||||
**Parallelizable with:** 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SiteAuditRetentionService.cs` (`RunLoopAsync` :62-91)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SiteAuditRetentionServiceTests.cs` (extend; the `RecordingSiteAuditQueue` stub gains a block-until-cancelled mode)
|
||||
|
||||
1. Failing test:
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task StopAsync_MidPurge_CompletesCleanly_WithoutSurfacingCancellation()
|
||||
{
|
||||
var queue = new RecordingSiteAuditQueue { BlockUntilCancelled = true }; // PurgeExpiredAsync: await Task.Delay(Timeout.Infinite, ct)
|
||||
var options = Options.Create(new SiteAuditRetentionOptions
|
||||
{ RetentionDays = 7, PurgeInterval = TimeSpan.FromHours(24), InitialDelay = TimeSpan.Zero });
|
||||
using var svc = new SiteAuditRetentionService(queue, options, NullLogger<SiteAuditRetentionService>.Instance);
|
||||
await svc.StartAsync(CancellationToken.None);
|
||||
await WaitUntilAsync(() => queue.PurgeCalls.Count >= 1, TimeSpan.FromSeconds(5));
|
||||
|
||||
// Today: FAIL — the OCE SafePurgeAsync deliberately rethrows escapes RunLoopAsync
|
||||
// (the two awaits at :76/:89 sit outside any try/catch), cancels _loop, and
|
||||
// StopAsync returns that canceled task straight to the host (:122-126), whose
|
||||
// await throws TaskCanceledException into every shutdown log (R7).
|
||||
await svc.StopAsync(CancellationToken.None); // must NOT throw
|
||||
}
|
||||
```
|
||||
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter StopAsync_MidPurge` — expect FAIL (`TaskCanceledException`).
|
||||
3. Implementation — catch OCE at the loop boundary (the report's preferred alignment with the sibling `SiteAuditBacklogReporter` pattern), leaving `SafePurgeAsync`'s deliberate rethrow (:106-110) untouched so a mid-purge shutdown still aborts the purge promptly:
|
||||
|
||||
```csharp
|
||||
private async Task RunLoopAsync(CancellationToken ct)
|
||||
{
|
||||
try
|
||||
{
|
||||
// ... existing body verbatim (initial delay, first purge, delay/purge loop) ...
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
// Shutdown landed mid-purge: SafePurgeAsync rethrows OCE by design so the purge
|
||||
// aborts promptly, but the loop task must complete CLEANLY — StopAsync hands
|
||||
// _loop straight to the host, and a canceled task there is shutdown-log noise
|
||||
// (arch-review 04 round 2, R7). Cancellation here IS the clean exit.
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter SiteAuditRetentionServiceTests` — PASS (the two existing tick tests must stay green).
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SiteAuditRetentionService.cs tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SiteAuditRetentionServiceTests.cs && git commit -m "fix(audit-log): retention loop absorbs shutdown cancellation instead of faulting host StopAsync (plan R2-04 T12)"`
|
||||
|
||||
---
|
||||
|
||||
## Dependencies on other plans
|
||||
|
||||
- None of the round-1 plans have open tasks (the whole arch-review initiative closed 191/192 on 2026-07-10); no cross-plan file contention is expected.
|
||||
- Task 10 is this plan's **only EF migration**. No other plan currently adds ConfigurationDatabase migrations, but if one appears, serialize on the model snapshot (same rule as round-1 Tasks 6→8).
|
||||
- The `IKpiHistoryRepository` addition (Task 2) is additive; any concurrently developed fake/stub implementing the interface must add the member — build the solution after Task 2 to surface stragglers immediately.
|
||||
|
||||
## Execution order
|
||||
|
||||
**P0 (the High + its enablers, in order):** 1 → 2 → 3 → 4 (fold fetch, watermark seam, sliced backfill, failover fast-path).
|
||||
**P1:** 5 → 6 → 7 (rate presentation chain), 10 → 11 (SiteCalls purge chain).
|
||||
**P2 (low-severity cleanup):** 8, 9, 12 — broadly parallel per each task's "Parallelizable with" contract (watch the two serialization lanes: `KpiHistoryRepository.cs` = 1 → 2 → 9; `Component-KpiHistory.md` = 4 → 7 → 9 in whatever order they run, never concurrently).
|
||||
Finish with `dotnet build ZB.MOM.WW.ScadaBridge.slnx` plus the targeted suites this plan touched — `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter "KpiHistoryRepositoryTests|SiteCallAuditRepositoryTests|SchemaConfigurationTests"` (infra up for the MSSQL `SkippableFact`s), `dotnet test tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests`, `dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests --filter "FullyQualifiedName~Kpi"`, `dotnet test tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests --filter "KpiHistoryQueryServiceTests|KpiTrendChartTests"`, `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests`, `dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests --filter SiteHealthKpiSampleSource`, and `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter "FullyQualifiedName~Site"` — before declaring the plan done.
|
||||
|
||||
## Findings Coverage
|
||||
|
||||
| # | Report finding (severity) | Task(s) |
|
||||
|---|---------------------------|---------|
|
||||
| R1 | Startup backfill folds the entire raw-retention window in one unbounded pass, re-run on every failover (High) | 2, 3, 4 |
|
||||
| R2 | Fold fetch is change-tracked — `AsNoTracking`/projection missing (Medium) | 1 |
|
||||
| R3 | Rate metrics change units ~60× across the raw/rollup routing boundary; bucketer re-introduces the last-value fold error on hourly Rate sums (Medium) | 5, 6, 7 |
|
||||
| R4 | `KpiMetricAggregationCatalog` re-creates the private-metric-literal drift hazard C4 fixed (Low) | 8 |
|
||||
| R5 | Overlapping-singleton fold race faults an entire fold batch — failure grain is the pass, not the row (Low) | 9 |
|
||||
| R6 | `PurgeTerminalAsync` on SiteCalls is the one remaining unbatched, unindexed year-scale maintenance DELETE (Medium) | 10, 11 |
|
||||
| R7 | `SiteAuditRetentionService.StopAsync` surfaces the loop's cancellation to the host (Low) | 12 |
|
||||
| S8 | Outbox repository dual SQLite/T-SQL dialect | No action — accepted round-1 deferral; round-2 re-verified the documented convention (`Component-ConfigurationDatabase.md:69`, "do not copy") |
|
||||
| S10 | `SqliteAuditWriter.Dispose` sync-over-async | No action — accepted round-1 won't-fix; round-2 re-verified |
|
||||
| P3 | AuditLog clustered key leads with random GUID | No action — accepted round-1 deferral, tracked benchmark follow-up (`Component-ConfigurationDatabase.md:70`) |
|
||||
| P4 (cadence) | Per-node KPI sampling cadence | No action — accepted round-1 deferral, tracked (`Component-ConfigurationDatabase.md:71`); the purge half shipped in round 1 |
|
||||
| P6 (paging) | Outbox offset→keyset page conversion | No action — accepted round-1 deferral, tracked (`Component-ConfigurationDatabase.md:71`); the KPI half shipped in round 1 |
|
||||
| C2 | `AuditLogRow` lives in ConfigurationDatabase, not Commons | No action — accepted round-1 deviation, documented (`Component-Commons.md:51`, "do not lift it into Commons") |
|
||||
| C3 | Mixed timestamp CLR types | No action — accepted round-1 convention note (`Component-ConfigurationDatabase.md:68`) |
|
||||
| U3 | Hash-chain tamper evidence / Parquet archival | No action — unchanged v1.x deferral; round-2 confirms no drift |
|
||||
| U5 (residue) | AuditLog `PullAuditEvents` keyset upgrade | No action — accepted round-1 deferral, tracked (`Component-ConfigurationDatabase.md:71`); idempotent on `EventId`, lower urgency; round-2 lists it only so the tracker entry doesn't age out |
|
||||
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"planPath": "archreview/plans/PLAN-R2-04-data-audit-backbone.md",
|
||||
"tasks": [
|
||||
{ "id": 1, "subject": "Task 1: Untracked, projected fold fetch (R2)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 2, "subject": "Task 2: Rollup watermark seam — GetLatestRollupHourAsync (R1, part 1)", "status": "pending", "blockedBy": [1] },
|
||||
{ "id": 3, "subject": "Task 3: Slice the rollup backfill into bounded day windows (R1, part 2)", "status": "pending", "blockedBy": [2] },
|
||||
{ "id": 4, "subject": "Task 4: Backfill failover fast-path via the rollup watermark + doc correction (R1, part 3)", "status": "pending", "blockedBy": [2, 3] },
|
||||
{ "id": 5, "subject": "Task 5: Per-metric reduction in the bucketer — sum-per-bucket for Rate series (R3, part 1)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 6, "subject": "Task 6: Catalog-driven aggregation at the query-service boundary (R3, part 2)", "status": "pending", "blockedBy": [5] },
|
||||
{ "id": 7, "subject": "Task 7: Truthful trend presentation — chart doc + Component-KpiHistory.md (R3, part 3)", "status": "pending", "blockedBy": [6] },
|
||||
{ "id": 8, "subject": "Task 8: Close the catalog metric-literal drift hazard (R4)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 9, "subject": "Task 9: Classify the failover fold-race failure grain (R5)", "status": "pending", "blockedBy": [1, 2] },
|
||||
{ "id": 10, "subject": "Task 10: SiteCalls filtered terminal index (migration) (R6, part 1)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 11, "subject": "Task 11: Time-sliced SiteCalls terminal purge (R6, part 2)", "status": "pending", "blockedBy": [10] },
|
||||
{ "id": 12, "subject": "Task 12: Retention-service shutdown no longer surfaces cancellation (R7)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 13, "subject": "Final verification: solution build + targeted suites this plan touched (infra up for MSSQL SkippableFacts)", "status": "pending", "blockedBy": [4, 7, 8, 9, 11, 12] }
|
||||
],
|
||||
"lastUpdated": "2026-07-13T03:25:01Z"
|
||||
}
|
||||
@@ -0,0 +1,429 @@
|
||||
# PLAN-R2-05 — Templates, Deployment & Transport Round-2 Fix Implementation Plan
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
|
||||
|
||||
**Goal:** Close the six NEW findings (N1–N6) in the round-2 report `archreview/05-templates-deployment-transport.md` (2026-07-12, re-review at HEAD `8c888f13`): the Expression-trigger Roslyn-compile leftover on read-only staleness paths (N1 — the sole Medium, the surviving member class of round-1 #12), the first-error-only trigger syntax message (N2 — the trigger twin of the T24 fix), the change-bus publisher skipping `Add` resolutions (N3), silently-inert locked-member instance overrides on import (N4), the instance-alarm-override gap in the import trust gate (N5), and the unvalidated `MaxConcurrentImportSessions` knob (N6). All round-1 accepted deferrals (incl. the #15 `ReadManifestAsync` residue) stay deferred — coverage rows only, no tasks.
|
||||
|
||||
**Architecture:** Two independent lanes. **TemplateEngine lane (N1+N2):** `ValidationService.ValidateExpressionTriggers` today runs `ScriptTrustValidator.FindViolations` + `RoslynScriptCompiler.Compile` per Expression-triggered script/alarm unconditionally and uncached (`ValidationService.cs:131, 531-544`) — the fix mirrors round-1 #12 exactly: memoise the verdict in `ScriptCompileVerdictCache` (whose key must first gain a **globals-surface discriminator** — it is keyed on code alone at `ScriptCompileVerdictCache.cs:53`, sound only while `ScriptCompiler`/`ScriptCompileSurface` is its sole writer; a `TriggerCompileSurface` verdict is not interchangeable), then gate the syntax/compile stage behind the same `validateScriptCompilation` flag that already carries `false` from `GetDeploymentComparisonAsync` and `StaleInstanceProbe` through `FlatteningPipeline` — the blank-expression check and attribute-reference scan stay unconditional, and the deploy gate keeps the default (`true`). **Transport lane (N3+N4+N5):** three surgical `BundleImporter` changes — publish `ScriptArtifactsChanged` for non-Skip resolutions including `Add` (`:1798-1803`); extend `EnumerateTrustGatedScripts` (`:957-1001`, the single enumerator both the apply-time Pass-0 gate at `:4309` and the preview gate at `:849` share, so one change covers both) to `InstanceAlarmOverrideDto.TriggerConfigurationOverride` expression bodies; and emit best-effort warnings (`ConflictKind.Warning` at preview + `ImportResult.Warnings` at apply) when an instance override targets a locked template member, WITHOUT changing what gets written or how the flattener resolves it. N6 is a one-line `RequireThat` in `TransportOptionsValidator`.
|
||||
|
||||
**Tech Stack:** C#/.NET, EF Core (in-memory + MSSQL integration fixtures), Roslyn (`Microsoft.CodeAnalysis.CSharp.Scripting`), xUnit. Build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Test per project: `dotnet test tests/<project>` — targeted filters only per task; no full-suite runs until the plan's terminal verification.
|
||||
|
||||
## Parallelization
|
||||
|
||||
**Concurrent lanes (dispatch as separate implementers):**
|
||||
- **TemplateEngine lane (serialize — shared `ValidationService.cs` + `ScriptCompilerTests.cs`):** T1 → T2 → T3 → T4.
|
||||
- **Transport `BundleImporter.cs` mutex (serialize):** T5 → T6 → T7.
|
||||
- **Free / file-disjoint:** T8 (`TransportOptionsValidator.cs`) runs concurrently with everything.
|
||||
- **Last:** T9 (docs sweep, blockedBy all).
|
||||
|
||||
**Cross-plan cautions:** T5 is the **publisher half** of N3 only — the subscriber side (Inbound API wiring + the wrong Host comment) is owned by **PLAN-R2-06**. The publisher fix is independent and self-contained (the plan-06 consumer already self-heals by content comparison, `InboundScriptExecutor.cs:370-404`), so there is **no blockedBy across plans** — either plan can land first; the coverage table records the split so both halves land coherently.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Full violation/error lists in the trigger-expression syntax check (N2)
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~3 min
|
||||
**Parallelizable with:** 5, 6, 7, 8
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Validation/ValidationService.cs` (`CheckExpressionSyntax` :536, :541)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/Validation/ScriptCompilerTests.cs` (extend — the existing `CheckExpressionSyntax_*` facts live here, :134-148)
|
||||
|
||||
1. Write failing tests:
|
||||
```csharp
|
||||
[Fact]
|
||||
public void CheckExpressionSyntax_MultipleForbiddenApis_ReportsAll()
|
||||
{
|
||||
var error = ValidationService.CheckExpressionSyntax(
|
||||
"System.IO.File.Exists(\"x\") && System.Diagnostics.Process.GetProcesses().Length > 0");
|
||||
Assert.NotNull(error);
|
||||
Assert.Contains("System.IO", error); // FAILS today: only violations[0] is surfaced
|
||||
Assert.Contains("Process", error);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void CheckExpressionSyntax_MultipleCompileErrors_ReportsAll()
|
||||
{
|
||||
var error = ValidationService.CheckExpressionSyntax("NoSuchThingA > 1 && NoSuchThingB < 2");
|
||||
Assert.NotNull(error);
|
||||
Assert.Contains("NoSuchThingA", error); // FAILS today: only errors[0] is surfaced
|
||||
Assert.Contains("NoSuchThingB", error);
|
||||
}
|
||||
```
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests --filter FullyQualifiedName~ScriptCompilerTests` → expect **FAIL** (second violation/error missing from the message).
|
||||
3. Implement — the exact `string.Join` shape Task 24 gave `ScriptCompiler.TryCompile` (`ScriptCompiler.cs:50, 55`), applied to the trigger twin at `ValidationService.cs:534-544`:
|
||||
```csharp
|
||||
var violations = ScriptTrustValidator.FindViolations(expression);
|
||||
if (violations.Count > 0)
|
||||
return $"uses forbidden API: {string.Join("; ", violations)}";
|
||||
|
||||
var errors = RoslynScriptCompiler.Compile(expression, typeof(TriggerCompileSurface));
|
||||
if (errors.Count > 0)
|
||||
return $"is not a valid expression: {string.Join("; ", errors)}";
|
||||
```
|
||||
An operator fixing a multi-error expression sees every finding in one deploy round-trip, matching the "Report ALL violations, not just the first" comment `ScriptCompiler.cs:46-47` already carries.
|
||||
4. Run → expect **PASS**. Re-run the whole filter to confirm the existing `CheckExpressionSyntax_*` facts still pass.
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Validation/ValidationService.cs tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/Validation/ScriptCompilerTests.cs && git commit -m "fix(template-engine): trigger-expression syntax check reports ALL violations/compile errors, not just the first (plan R2-05 T1)"`
|
||||
|
||||
### Task 2: Add a globals-surface discriminator to the verdict-cache key (N1 part 1)
|
||||
|
||||
**Classification:** high-risk (keying of a cache that stores script-trust verdicts — a cross-surface verdict reuse would return a stale "clean" for code never vetted against that surface)
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** 5, 6, 7, 8 (blockedBy 1 — shared `ScriptCompilerTests.cs`)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Validation/ScriptCompileVerdictCache.cs` (`GetOrAdd` :51-69 gains a leading `string surface` parameter; key at :53 becomes `surface + ":" + hash`)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Validation/ScriptCompiler.cs` (:42 — pass `nameof(ScriptCompileSurface)`)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/Validation/ScriptCompilerTests.cs` (extend)
|
||||
|
||||
1. Write failing test (compile-red first — the signature changes — then behavioural red):
|
||||
```csharp
|
||||
[Fact]
|
||||
public void GetOrAdd_SameCodeDifferentSurface_ComputesSeparateVerdicts()
|
||||
{
|
||||
ScriptCompileVerdictCache.Clear();
|
||||
var a = ScriptCompileVerdictCache.GetOrAdd("SurfaceA", "return 1;", () => (true, null));
|
||||
var hitsAfterA = ScriptCompileVerdictCache.Hits;
|
||||
var b = ScriptCompileVerdictCache.GetOrAdd("SurfaceB", "return 1;", () => (false, "err"));
|
||||
|
||||
Assert.True(a.Ok);
|
||||
Assert.False(b.Ok); // code-only key would return SurfaceA's verdict
|
||||
Assert.Equal(hitsAfterA, ScriptCompileVerdictCache.Hits); // no false cross-surface hit
|
||||
Assert.Equal(2, ScriptCompileVerdictCache.Count);
|
||||
}
|
||||
```
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests --filter FullyQualifiedName~ScriptCompilerTests` → expect **FAIL** (compile error on the new parameter — that counts; fix the signature, then the behavioural assertions go red under a code-only key).
|
||||
3. Implement:
|
||||
- `GetOrAdd(string surface, string code, Func<(bool Ok, string? Error)> factory)`; key = `surface + ":" + Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(code)))`. Update the class doc-comment: *the verdict is a pure function of code + policy + globals surface, so the surface is part of the key — a trigger expression valid against `TriggerCompileSurface` is NOT interchangeable with a `ScriptCompileSurface` script-body verdict* (the exact hazard the round-2 report calls out for N1).
|
||||
- `ScriptCompiler.TryCompile` passes `nameof(ScriptCompileSurface)` at :42. No behaviour change for script bodies — same single writer, now under an explicit key segment.
|
||||
4. Run the filter → expect **PASS** (including the pre-existing `TryCompile_SameCodeTwice_SecondCallIsCacheHit` cache facts at :157-171 — they exercise one surface and must be unaffected).
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Validation/ScriptCompileVerdictCache.cs src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Validation/ScriptCompiler.cs tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/Validation/ScriptCompilerTests.cs && git commit -m "fix(template-engine): verdict-cache key gains the globals-surface discriminator — cross-surface verdict reuse structurally impossible (plan R2-05 T2)"`
|
||||
|
||||
### Task 3: Cache Expression-trigger verdicts under the trigger surface key (N1 part 2)
|
||||
|
||||
**Classification:** high-risk (trust-gate path — the trigger syntax check is a forbidden-API verdict; a caching bug here weakens the gate)
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** 5, 6, 7, 8 (blockedBy 2 — same files)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Validation/ValidationService.cs` (`CheckExpressionSyntax` :531-544 routes through `ScriptCompileVerdictCache`)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/Validation/ScriptCompilerTests.cs` (extend)
|
||||
|
||||
1. Write failing test plus the cross-surface lock-in negative:
|
||||
```csharp
|
||||
[Fact]
|
||||
public void CheckExpressionSyntax_SameExpressionTwice_SecondCallIsCacheHit()
|
||||
{
|
||||
ScriptCompileVerdictCache.Clear();
|
||||
ValidationService.CheckExpressionSyntax("Attributes[\"Temp\"] != null");
|
||||
var hitsBefore = ScriptCompileVerdictCache.Hits;
|
||||
var error = ValidationService.CheckExpressionSyntax("Attributes[\"Temp\"] != null");
|
||||
Assert.Null(error);
|
||||
Assert.Equal(hitsBefore + 1, ScriptCompileVerdictCache.Hits); // FAILS today: no cache use
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void CheckExpressionSyntax_ScriptSurfaceVerdict_NotReusedForTriggerSurface()
|
||||
{
|
||||
// "Notify != null" resolves on ScriptCompileSurface (Notify is a script global,
|
||||
// ScriptCompileSurface.cs:53) but NOT on TriggerCompileSurface — a shared
|
||||
// code-only cache entry would wrongly report the trigger expression clean.
|
||||
ScriptCompileVerdictCache.Clear();
|
||||
Assert.True(new ScriptCompiler().TryCompile("Notify != null", "S").IsSuccess); // warms the SCRIPT-surface entry
|
||||
var error = ValidationService.CheckExpressionSyntax("Notify != null");
|
||||
Assert.NotNull(error); // green today (no cache), and MUST STAY green after T3 — the T2 key makes it structural
|
||||
}
|
||||
```
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests --filter FullyQualifiedName~ScriptCompilerTests` → expect **FAIL** (the cache-hit fact; the cross-surface fact is the lock-in guard that must never flip).
|
||||
3. Implement — wrap the T1-joined factory in the surface-keyed cache:
|
||||
```csharp
|
||||
internal static string? CheckExpressionSyntax(string expression)
|
||||
{
|
||||
// Memoise the verdict under the TRIGGER surface key (see ScriptCompileVerdictCache):
|
||||
// the verdict is a pure function of expression + policy + TriggerCompileSurface, and
|
||||
// it is the exact hot-path cost N1 flagged — every staleness sweep / import probe of
|
||||
// an Expression-triggered config re-ran a full trust compilation + script compile.
|
||||
var (_, error) = ScriptCompileVerdictCache.GetOrAdd(nameof(TriggerCompileSurface), expression, () =>
|
||||
{
|
||||
var violations = ScriptTrustValidator.FindViolations(expression);
|
||||
if (violations.Count > 0)
|
||||
return (false, $"uses forbidden API: {string.Join("; ", violations)}");
|
||||
|
||||
var errors = RoslynScriptCompiler.Compile(expression, typeof(TriggerCompileSurface));
|
||||
if (errors.Count > 0)
|
||||
return (false, $"is not a valid expression: {string.Join("; ", errors)}");
|
||||
|
||||
return (true, (string?)null);
|
||||
});
|
||||
return error;
|
||||
}
|
||||
```
|
||||
The stored error is already entity-name-free — `CheckExpressionTrigger` formats the entity name in at :449-451, matching the cache's name-free contract.
|
||||
4. Run the filter → expect **PASS** (both new facts + all pre-existing `CheckExpressionSyntax_*`/`TryCompile_*` facts).
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Validation/ValidationService.cs tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/Validation/ScriptCompilerTests.cs && git commit -m "perf(template-engine): cache Expression-trigger compile verdicts under the trigger-surface key — unchanged expressions compile once per process (plan R2-05 T3)"`
|
||||
|
||||
### Task 4: Skip the trigger syntax check on read-only staleness/comparison paths (N1 part 3)
|
||||
|
||||
**Classification:** standard (read paths only; the deploy gate keeps the authoritative compile by default — the same shape round-1 #12/T16 shipped for script bodies)
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** 5, 6, 7, 8 (blockedBy 3 — same files)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Validation/ValidationService.cs` (`Validate` :131 threads the flag; `ValidateExpressionTriggers` :370-400 + `CheckExpressionTrigger` :420-463 gain the gate; `validateScriptCompilation` doc-comment :98-105 updated)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/Validation/ValidationServiceTests.cs` (extend)
|
||||
|
||||
1. Write failing tests (build the config the way the file's existing facts do — a `FlattenedConfiguration` with one script whose `TriggerType = "Expression"` and `TriggerConfiguration = """{"expression":"this is not C# (("}"""`):
|
||||
```csharp
|
||||
[Fact]
|
||||
public void Validate_SkipCompilation_SkipsExpressionTriggerSyntaxCheck()
|
||||
{
|
||||
var config = ConfigWithExpressionTriggerScript("this is not C# ((");
|
||||
var gated = new ValidationService().Validate(config, validateScriptCompilation: false);
|
||||
Assert.DoesNotContain(gated.Errors, e => e.Message.Contains("failed validation")); // FAILS today
|
||||
var full = new ValidationService().Validate(config);
|
||||
Assert.Contains(full.Errors, e => e.Message.Contains("failed validation")); // deploy gate unchanged
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Validate_SkipCompilation_StillChecksExpressionAttributeReferences()
|
||||
{
|
||||
// expression "Attributes[\"Ghost\"] != null" referencing a missing attribute:
|
||||
// the reference scan is cheap string work and MUST survive the gate.
|
||||
var config = ConfigWithExpressionTriggerScript("Attributes[\"Ghost\"] != null");
|
||||
var gated = new ValidationService().Validate(config, validateScriptCompilation: false);
|
||||
Assert.Contains(gated.Errors, e => e.Message.Contains("Ghost"));
|
||||
}
|
||||
```
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests --filter FullyQualifiedName~ValidationServiceTests` → expect **FAIL** (the gated run still emits the syntax error).
|
||||
3. Implement:
|
||||
- `ValidateExpressionTriggers(FlattenedConfiguration configuration, bool validateExpressionSyntax = true)` — additive default parameter, so every existing direct caller/test stays source-compatible.
|
||||
- `Validate` passes `validateScriptCompilation` through at :131 — the same flag `FlatteningPipeline` already threads `false` into from `GetDeploymentComparisonAsync` (`DeploymentService.cs:690`) and `StaleInstanceProbe` (`StaleInstanceProbe.cs:37`), so **no DeploymentManager/Transport change is needed**: both read paths stop paying the per-expression trust-compilation + compile the moment this lands.
|
||||
- Thread the flag into `CheckExpressionTrigger`; when `false`, skip ONLY the `CheckExpressionSyntax` call (:446-452) — the blank-expression warning/error (:432-444) and the attribute-reference scan (:454-462) still run, per the report's "the attribute-reference scan can stay".
|
||||
- Extend the `validateScriptCompilation` doc-comment (:98-105): the flag now gates both the script-body compile stage AND the Expression-trigger syntax/compile check; read-only callers skip both, the deploy gate runs both.
|
||||
4. Run the filter, then the full TemplateEngine project (`dotnet test tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests`) → expect **PASS**, no regressions.
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Validation/ValidationService.cs tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/Validation/ValidationServiceTests.cs && git commit -m "perf(template-engine): read-only staleness/comparison paths skip Expression-trigger compiles — deploy gate remains the authoritative check (plan R2-05 T4)"`
|
||||
|
||||
### Task 5: Publish `ScriptArtifactsChanged` for `Add` resolutions too (N3)
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~3 min
|
||||
**Parallelizable with:** 1, 2, 3, 4, 8
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs` (`PublishScriptArtifactChanges` filter :1802 + the doc-comment :1768-1776)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests/Import/BundleImporterApplyTests.cs` (extend — the `RecordingScriptArtifactChangeBus` fixture and the `#05-T14` publish fact at :1336-1455 are already there)
|
||||
|
||||
1. Write failing test alongside `ApplyAsync_publishes_ScriptArtifactsChanged_per_kind_after_commit` (:1339):
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task ApplyAsync_publishes_ScriptArtifactsChanged_for_Add_resolutions()
|
||||
{
|
||||
// ApiMethod imported as Add into a target where the name does not exist.
|
||||
// Delete-then-reimport-as-Add is the N3 edge: a node can still hold a
|
||||
// _knownBadMethods / compiled-handler entry under that very name, so Adds
|
||||
// must notify too (over-approximation is safe per the bus contract).
|
||||
var result = await ApplyBundleWithApiMethodAsync(action: ResolutionAction.Add);
|
||||
|
||||
var notification = Assert.Single(_artifactBus.Received,
|
||||
n => n.ArtifactKind == ScriptArtifactKinds.ApiMethod);
|
||||
Assert.Contains("DelmiaRecipeDownload", notification.Names); // FAILS today: Adds are filtered out
|
||||
}
|
||||
```
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests --filter FullyQualifiedName~BundleImporterApplyTests` → expect **FAIL** (no ApiMethod notification — `names.Count == 0` short-circuits).
|
||||
3. Implement — one condition at :1802 plus the comment that justified the old behaviour:
|
||||
```csharp
|
||||
.Where(r => string.Equals(r.EntityType, entityType, StringComparison.Ordinal)
|
||||
&& r.Action is not ResolutionAction.Skip)
|
||||
```
|
||||
Rewrite the doc-comment (:1768-1776): *publishes for every non-Skip resolution — Overwrite, Rename, AND Add. An Add is not "nothing cached yet": an artifact deleted on the target and re-imported as Add under the same name can still have a stale compiled-handler/`_knownBadMethods` entry on any node. Over-approximation stays safe (the bus is advisory, at-least-once; consumers self-heal), which is the contract's stated design.* Keep the Rename → `RenameTo` name projection at :1803 as-is.
|
||||
4. Run the filter → expect **PASS** (including the existing per-kind Overwrite fact and the failed-apply-publishes-nothing fact — the post-commit placement at :1340 is untouched).
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests/Import/BundleImporterApplyTests.cs && git commit -m "fix(transport): script-artifact change publisher covers Add resolutions — delete-then-reimport no longer starves non-self-healing subscribers (plan R2-05 T5)"`
|
||||
|
||||
### Task 6: Trust-gate instance alarm-override trigger expressions (N5)
|
||||
|
||||
**Classification:** high-risk (script trust gate — 5th `ScriptTrustValidator` call-site completeness; security-adjacent)
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1, 2, 3, 4, 8 (blockedBy 5 — `BundleImporter.cs` mutex)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs` (`EnumerateTrustGatedScripts` :957-1001 gains the instance pass; `ExtractTriggerExpression` :1009-1033 refactored to expose a JSON-only `ExtractExpressionBody`)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests/SemanticValidatorImportTests.cs` (extend — sits beside the template twin `Apply_AlarmExpressionTriggerWithForbiddenApi_HardBlocks` at :371)
|
||||
|
||||
**Severity decision (recorded here, per the review convention):** instance-override trigger findings are **hard errors**, matching every other trust-gate surface. The advisory-vs-hard severity split is a property of the *name-resolution heuristic* only (false-positive-prone, so template findings are warnings — `:819, :842-846`); the trust gate's semantic verdict is authoritative with no false-positive channel, and Task 20 fixed its severity as *"a HARD error for all kinds"* (`:4301-4308`) — template script bodies and template trigger expressions already hard-block today. An instance override is the same executable surface (it *replaces* the template's trigger configuration in the flattened config and compiles/executes at the site), so it takes the same severity. The deploy gate (`ValidateExpressionTriggers` over the flattened, post-override config) stays the authoritative backstop — this closes the import-*review* completeness gap so the operator learns in the wizard, not at deploy time.
|
||||
|
||||
1. Write failing tests (negative tests required for a trust-gate change):
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task Apply_InstanceAlarmOverrideExpressionWithForbiddenApi_HardBlocks()
|
||||
{
|
||||
// Bundle: template with an Expression-triggered alarm + an instance whose
|
||||
// InstanceAlarmOverrideDto.TriggerConfigurationOverride carries
|
||||
// {"expression":"System.Diagnostics.Process.GetProcesses().Length > 0"}.
|
||||
var ex = await Assert.ThrowsAsync<SemanticValidationException>(
|
||||
() => ApplyBundleAsync(bundle)); // FAILS today: imports clean
|
||||
Assert.Contains(ex.Errors, e => e.Contains("Process") && e.Contains("trigger override"));
|
||||
// Rollback contract: nothing persisted (mirror the template-twin's assertions).
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Preview_InstanceAlarmOverrideExpressionWithForbiddenApi_SurfacesBlocker()
|
||||
{
|
||||
var preview = await PreviewBundleAsync(bundle);
|
||||
Assert.Contains(preview.Blockers, b =>
|
||||
b.Kind == ConflictKind.Blocker && b.EntityType == "Instance"
|
||||
&& b.BlockerReason!.Contains("trust violation")); // FAILS today
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Apply_InstanceAlarmOverrideExpression_Clean_Imports()
|
||||
{
|
||||
// {"expression":"Attributes[\"Temp\"] != null"} → import succeeds, override row persisted verbatim.
|
||||
}
|
||||
```
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests --filter FullyQualifiedName~SemanticValidatorImportTests` → expect **FAIL** (both forbidden-API cases import/preview clean).
|
||||
3. Implement:
|
||||
- Split `ExtractTriggerExpression` (:1009-1033): the trigger-type guard stays in `ExtractTriggerExpression`, the JSON `{"expression":"…"}` parse moves to a new `private static string? ExtractExpressionBody(string? triggerConfigJson)` it delegates to.
|
||||
- Append the instance pass to `EnumerateTrustGatedScripts` after the ApiMethod loop (:993-1000):
|
||||
```csharp
|
||||
foreach (var i in content.Instances)
|
||||
{
|
||||
if (IsSkipResolution(resolutionMap, "Instance", i.UniqueName)) continue;
|
||||
foreach (var o in i.AlarmOverrides)
|
||||
{
|
||||
// The DTO carries no trigger type (the type lives on the template
|
||||
// alarm), so gate ANY override config carrying an {"expression":...}
|
||||
// string body: if the overridden alarm is not Expression-triggered
|
||||
// the body never executes, but vetting it anyway is fail-safe — the
|
||||
// structured (HiLo/threshold) configs have no "expression" key, so
|
||||
// there is no false-positive channel.
|
||||
var expr = ExtractExpressionBody(o.TriggerConfigurationOverride);
|
||||
if (!string.IsNullOrEmpty(expr))
|
||||
{
|
||||
yield return ("Instance", i.UniqueName,
|
||||
$"{o.AlarmCanonicalName} (alarm trigger override expression)", expr);
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
- No gate-side change needed: the apply-time Pass 0 (:4309) and the preview gate (:849) both iterate this enumerator, so one edit covers both, including the fail-closed validator-throw handling each already has. Update the enumerator's doc-comment (:946-956) to name instance alarm-override expressions in the covered-surface list.
|
||||
4. Run the filter, then `dotnet test tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests --filter FullyQualifiedName~BundleImporterPreviewTests` (preview parity) → expect **PASS**.
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests/SemanticValidatorImportTests.cs && git commit -m "fix(security): import trust gate covers instance alarm-override trigger expressions — 5th call site is surface-complete (plan R2-05 T6)"`
|
||||
|
||||
### Task 7: Warn when import persists overrides on locked template members (N4)
|
||||
|
||||
**Classification:** standard (warning-only UX; no write-path or flattener behaviour change — the flattener already drops locked-member overrides, `FlatteningService.cs:319, :337, :815`)
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1, 2, 3, 4, 8 (blockedBy 6 — `BundleImporter.cs` mutex)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs` (new best-effort lock scan called from `RunSemanticValidationAsync` (:4293, appending to `warnings`) and from `DetectBlockersAsync` (:670, emitting `ConflictKind.Warning` rows); generalize the apply-side warning log wording at :1204-1208)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests/Import/SiteInstanceImportTests.cs` (extend)
|
||||
|
||||
1. Write failing tests:
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task Import_OverrideOnLockedAttribute_EmitsWarning_ImportStillSucceeds()
|
||||
{
|
||||
// Template attribute "SetPoint" IsLocked=true; instance carries an
|
||||
// InstanceAttributeOverrideDto for "SetPoint".
|
||||
var result = await ApplyBundleAsync(bundle);
|
||||
Assert.Contains(result.Warnings, w => w.Contains("SetPoint") && w.Contains("locked")); // FAILS today: silent
|
||||
// Behaviour unchanged: the row IS still written (bundle fidelity) — the
|
||||
// flattener is what ignores it; assert the persisted override row exists.
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Preview_OverrideOnLockedNativeAlarmSource_ShowsWarningRow()
|
||||
{
|
||||
var preview = await PreviewBundleAsync(bundle);
|
||||
Assert.Contains(preview.Blockers, b =>
|
||||
b.Kind == ConflictKind.Warning && b.EntityType == "Instance"
|
||||
&& b.BlockerReason!.Contains("locked")); // FAILS today
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Import_OverrideOnUnlockedAttribute_NoLockWarning()
|
||||
```
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests --filter FullyQualifiedName~SiteInstanceImportTests` → expect **FAIL**.
|
||||
3. Implement one shared best-effort scan, `CollectLockedOverrideWarnings(BundleContentDto content, Dictionary<(string,string), ImportResolution>? resolutionMap, IReadOnlyList<Template> targetTemplates)`:
|
||||
- For each non-Skip instance, resolve its template by `dto.TemplateName` — bundle DTO first (the version about to be written), else the pre-existing target template (`GetAllTemplatesAsync` already Includes the child collections).
|
||||
- Locked-member name sets from that template's rows: attributes/alarms/native sources with `IsLocked == true` (bundle DTOs carry `IsLocked`; derived templates carry inherited placeholder rows, so own + inherited members match by name). Check `AttributeOverrides` (→ `FlatteningService.cs:319` drop), `AlarmOverrides` (→ `:337`), and `NativeAlarmSourceOverrides` (→ `:815`) — the alarm loop is the same pattern the report's two cited collections use, included for completeness.
|
||||
- Warning text: `Instance '{unique}' {kind} override '{name}' targets a LOCKED template member — the flattener ignores it, so this part of the bundle's instance config will never take effect.`
|
||||
- **Best-effort, documented:** overrides addressing composed path-qualified members (`y1.z.Val`) or derived-shadow locks (`LockedInDerived` on a base the placeholder row doesn't reflect) may not match a direct row — an unmatched name emits NO warning (never a false positive; the `ManagementActor` and flattener remain the enforcement points, `ManagementActor.cs:898, :978`).
|
||||
- Wire it twice: `RunSemanticValidationAsync` appends to `warnings` (they ride `validationWarnings` → `ImportResult.Warnings` at :1359 — generalize the "advisory template-script reference warning(s)" log wording at :1206-1208 to "advisory import warning(s)"); `DetectBlockersAsync` maps each to an `ImportPreviewItem` with `EntityType: "Instance"`, `Kind: ConflictKind.Warning` — the Task-19 machinery, so the wizard renders it with the existing Warning badge, no UI change.
|
||||
- Do **NOT** block the import and do **NOT** change what `PopulateInstanceChildren` writes (:3999-4029) — bundle fidelity keeps the row; the flattener stays the behavioural authority (binding scope ruling: no flattener change).
|
||||
4. Run the filter + `--filter FullyQualifiedName~BundleImporterPreviewTests` → expect **PASS**.
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests/Import/SiteInstanceImportTests.cs && git commit -m "fix(transport): warn on import of instance overrides targeting locked template members — inert rows no longer silent (plan R2-05 T7)"`
|
||||
|
||||
### Task 8: Validate `MaxConcurrentImportSessions` at startup (N6)
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~2 min
|
||||
**Parallelizable with:** 1, 2, 3, 4, 5, 6, 7 (file-disjoint with everything)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/TransportOptionsValidator.cs` (one `RequireThat` in `Validate` :25-66)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.Transport.Tests/TransportOptionsValidatorTests.cs` (extend — mirror `ZeroMaxBundleSizeMb_IsRejected` :34)
|
||||
|
||||
1. Write failing tests:
|
||||
```csharp
|
||||
[Fact]
|
||||
public void ZeroMaxConcurrentImportSessions_IsRejected()
|
||||
{
|
||||
var result = Validate(new TransportOptions { MaxConcurrentImportSessions = 0 });
|
||||
Assert.True(result.Failed);
|
||||
Assert.Contains("MaxConcurrentImportSessions", result.FailureMessage); // FAILS today: validates clean
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void NegativeMaxConcurrentImportSessions_IsRejected()
|
||||
```
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Transport.Tests --filter FullyQualifiedName~TransportOptionsValidatorTests` → expect **FAIL**.
|
||||
3. Implement — the knob its own fix wave added (`TransportOptions.cs:37`, enforced at `BundleSessionStore.cs:78-88` where a `0`/negative cap rejects **every** import forever with "Too many concurrent import sessions (0)"):
|
||||
```csharp
|
||||
builder.RequireThat(options.MaxConcurrentImportSessions > 0,
|
||||
$"ScadaBridge:Transport:MaxConcurrentImportSessions must be positive " +
|
||||
$"(was {options.MaxConcurrentImportSessions}); a zero/negative cap rejects every import session forever.");
|
||||
```
|
||||
4. Run the filter → expect **PASS** (`DefaultOptions_AreValid` must stay green — the default is 8).
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Transport/TransportOptionsValidator.cs tests/ZB.MOM.WW.ScadaBridge.Transport.Tests/TransportOptionsValidatorTests.cs && git commit -m "fix(transport): validate MaxConcurrentImportSessions at startup — a zero cap no longer bricks imports silently (plan R2-05 T8)"`
|
||||
|
||||
### Task 9: Design-doc sync sweep
|
||||
|
||||
**Classification:** small (docs only)
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** none (run last)
|
||||
**Files:**
|
||||
- Modify: `docs/requirements/Component-Transport.md`, `docs/requirements/Component-ScriptAnalysis.md`, `docs/requirements/Component-TemplateEngine.md`, `/Users/dohertj2/Desktop/ScadaBridge/CLAUDE.md` (component #24 blurb + Transport bullet)
|
||||
|
||||
No tests; concrete edits, then a stale-cross-reference sweep per repo convention:
|
||||
1. **Component-Transport.md:** (a) trust-gate section — the gated-surface list gains *instance alarm-override trigger expressions* (T6), same hard severity, with the severity-decision sentence (semantic verdict authoritative; the advisory split remains name-heuristic-only); (b) import-warning section — locked-member override warnings (T7): advisory `ConflictKind.Warning` at preview + `ImportResult.Warnings` at apply, best-effort by direct member name, rows still written, flattener remains the behavioural authority; (c) change-bus paragraph — publisher now notifies every non-Skip resolution including Add (T5), and note the subscriber side lives with the Inbound API (**PLAN-R2-06** owns the subscriber wiring + the stale Host comment); (d) options table — `MaxConcurrentImportSessions` is now startup-validated (T8). Leave the `ReadManifestAsync` deferral note (:92) untouched — still deferred.
|
||||
2. **Component-ScriptAnalysis.md:** fifth-call-site (Transport) surface list gains instance alarm-override trigger expressions; note the verdict cache is keyed by (globals surface, code hash) so script-body and trigger-expression verdicts never cross (T2/T3).
|
||||
3. **Component-TemplateEngine.md:** validation-pipeline section — `validateScriptCompilation: false` now also skips the Expression-trigger syntax/compile check on read-only staleness/comparison paths (blank + attribute-reference checks still run; deploy gate authoritative, T4); trigger syntax findings report all violations/errors (T1).
|
||||
4. **CLAUDE.md:** component #24 blurb / Transport Key-Design-Decisions bullet — extend the script-trust-gate sentence: Expression-trigger gating covers template scripts, alarms, **and instance alarm-override expressions**. No `../scadaproj/CLAUDE.md` change (no wire-relationship/stack change) — verify and note in the commit message.
|
||||
5. `git diff` review, then commit: `git add docs/requirements/Component-Transport.md docs/requirements/Component-ScriptAnalysis.md docs/requirements/Component-TemplateEngine.md CLAUDE.md && git commit -m "docs: sync Transport/ScriptAnalysis/TemplateEngine specs + CLAUDE.md with the round-2 fix wave (plan R2-05 T9)"`
|
||||
|
||||
---
|
||||
|
||||
## Dependencies on other plans
|
||||
|
||||
- **PLAN-R2-06 (Edge Integrations round 2):** owns the **subscriber side** of the change bus (Inbound API wiring + the wrong Host comment). T5's publisher fix is independent — the shipped plan-06 consumer self-heals by content comparison (`InboundScriptExecutor.cs:370-404`), so no cross-plan `blockedBy` exists in either direction; the two halves land coherently in any order. R2-06 should read T5's updated publisher doc-comment (Adds now publish) when wiring any non-self-healing subscriber.
|
||||
- **No other cross-plan edges.** N1's gating rides the `validateScriptCompilation` flag PLAN-05 T16 already threads from DeploymentManager — no DeploymentManager file is touched.
|
||||
|
||||
## Execution order
|
||||
|
||||
**P0 (start immediately, in parallel):** Task 1 (TemplateEngine lane opener), Task 5 (BundleImporter lane opener), Task 8 (free).
|
||||
**Lane A (serialize — shared files):** 1 → 2 → 3 → 4.
|
||||
**Lane B (serialize — `BundleImporter.cs` mutex):** 5 → 6 → 7.
|
||||
**Last:** 9 (docs sweep), then terminal verification: `dotnet build ZB.MOM.WW.ScadaBridge.slnx` + `dotnet test tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests && dotnet test tests/ZB.MOM.WW.ScadaBridge.Transport.Tests && dotnet test tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests && dotnet test tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests` (DeploymentManager consumes the `Validate` flag) before declaring the plan done.
|
||||
|
||||
## Findings Coverage
|
||||
|
||||
| Report finding | Severity | Task(s) |
|
||||
|---|---|---|
|
||||
| N1 — Expression-trigger validation Roslyn-compiles on read-only staleness paths, uncached; any shared cache must key by globals surface | Medium (perf) | 2 (surface-discriminated cache key), 3 (cache trigger verdicts), 4 (skip on comparison/probe paths via the existing `validateScriptCompilation` flag) |
|
||||
| N2 — `CheckExpressionSyntax` surfaces only the first violation / first compile error | Low | 1 (the T24 `string.Join` fix mirrored onto the trigger twin) |
|
||||
| N3 — `PublishScriptArtifactChanges` skips `Add` resolutions | Low | 5 (publisher publishes every non-Skip resolution); **subscriber side (Inbound API wiring + wrong Host comment) → PLAN-R2-06** — publisher fix is independent, no cross-plan blockedBy |
|
||||
| N4 — Import persists silently-inert instance overrides on locked members | Low | 7 (`ConflictKind.Warning` preview rows + `ImportResult.Warnings` at apply; best-effort direct-member match; rows still written, flattener behaviour unchanged per ruling) |
|
||||
| N5 — Import trust gate misses `InstanceAlarmOverrideDto.TriggerConfigurationOverride` expression bodies | Low (sec) | 6 (enumerator covers instance overrides; **hard error** — trust-gate severity is uniform, the advisory split is name-heuristic-only; deploy gate stays authoritative backstop; negative tests included) |
|
||||
| N6 — `TransportOptionsValidator` misses `MaxConcurrentImportSessions` | Low (conv) | 8 (one `RequireThat` > 0) |
|
||||
| Docs drift from the above | — | 9 |
|
||||
| Accepted deferral — #11 `OperationLockManager` single-node in-memory | Medium (R1) | **Deferred** (unchanged) — invariant recorded `Component-DeploymentManager.md:94`; structural active-node gating owned by plans 01/07 |
|
||||
| Accepted deferral — #15 residue: `LoadAsync` manifest-peek (`ReadManifestAsync`) | Low (perf, R1) | **Deferred** (unchanged) — doc-acknowledged `Component-Transport.md:92`; pure optimisation (session cap already shipped, and N6/T8 now validates it) |
|
||||
| Accepted deferral — #16 import apply is one long EF transaction | Low (perf, R1) | **Deferred** (unchanged) — dominant cost removed by #12/N1; restructuring the rollback contract stays high-risk/low-gain |
|
||||
| Accepted deferral — U4: rename call-site rewriting, Transport-012 filter UI | — | **Deferred** (unchanged) — recorded limitations, `Component-Transport.md:354` |
|
||||
| Accepted deferral — U5: preview→apply optimistic concurrency window | — | **Deferred** (unchanged) — recorded decision, `Component-Transport.md:138`; version fields reserved (`ArtifactDiff.cs:33-37`) |
|
||||
| Accepted deferral — U6: three hand-maintained compile-surface mirrors | — | **Deferred** (unchanged) — guard tests exist; source-generator is an improvement, not a defect |
|
||||
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"planPath": "archreview/plans/PLAN-R2-05-templates-deployment-transport.md",
|
||||
"tasks": [
|
||||
{ "id": 1, "subject": "Task 1: Full violation/error lists in the trigger-expression syntax check (N2)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 2, "subject": "Task 2: Add a globals-surface discriminator to the verdict-cache key (N1 part 1)", "status": "pending", "blockedBy": [1] },
|
||||
{ "id": 3, "subject": "Task 3: Cache Expression-trigger verdicts under the trigger surface key (N1 part 2)", "status": "pending", "blockedBy": [2] },
|
||||
{ "id": 4, "subject": "Task 4: Skip the trigger syntax check on read-only staleness/comparison paths (N1 part 3)", "status": "pending", "blockedBy": [3] },
|
||||
{ "id": 5, "subject": "Task 5: Publish ScriptArtifactsChanged for Add resolutions too (N3)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 6, "subject": "Task 6: Trust-gate instance alarm-override trigger expressions (N5)", "status": "pending", "blockedBy": [5] },
|
||||
{ "id": 7, "subject": "Task 7: Warn when import persists overrides on locked template members (N4)", "status": "pending", "blockedBy": [6] },
|
||||
{ "id": 8, "subject": "Task 8: Validate MaxConcurrentImportSessions at startup (N6)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 9, "subject": "Task 9: Design-doc sync sweep", "status": "pending", "blockedBy": [1, 2, 3, 4, 5, 6, 7, 8] }
|
||||
],
|
||||
"lastUpdated": "2026-07-13T03:23:07Z"
|
||||
}
|
||||
@@ -0,0 +1,809 @@
|
||||
# PLAN-R2-06 — Edge Integrations Round-2 Fixes
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
|
||||
|
||||
**Goal:** Close every NEW finding in `archreview/06-edge-integrations.md` (round 2, 2026-07-12): wire the Inbound API compiled-handler cache as the real `IScriptArtifactChangeBus` consumer the Host comment and master tracker already claim exists (N1, the round-1 U2 residue) and sweep the false claims out of the docs; park (not retry-to-exhaustion) buffered ESG calls that fail deterministically with `ArgumentException` (N2); replace `SITE_UNREACHABLE` message-substring sniffing with a typed signal at the Ask boundary (N3); honor the declared response charset in the bounded body read (N4); and close the chunked-body bypass of the 415 guard (N5).
|
||||
|
||||
**Architecture:** The N1 consumer is a small `IHostedService` (`ScriptArtifactChangeSubscriber`) owned by the InboundAPI component and registered in `AddInboundAPI`: on start it calls `IScriptArtifactChangeBus.Subscribe` and, for every `ApiMethod` notification, calls the existing `InboundScriptExecutor.InvalidateMethod` seam (`InboundScriptExecutor.cs:136-140`) per name — the per-request revision check stays the correctness fallback per the contract's normative semantics (`docs/plans/2026-07-08-script-artifact-invalidation-contract.md` §4), so the subscriber is a fast-path, never a correctness dependency. The bus dependency is optional (`IScriptArtifactChangeBus? = null`, mirroring `BundleImporter.cs:134`) so site roles and plain test hosts no-op. N3 deliberately chooses the **typed-exception** form over an additive error-code field: unreachability manifests at the communication layer as `Akka.Actor.AskTimeoutException` (the site never answered — see `CentralCommunicationActor.HandleSiteEnvelope`'s no-ClusterClient path, which deliberately lets the Ask expire), so `RouteHelper` types it into `SiteUnreachableException` at the four router awaits and the message sniff (`RouteHelper.cs:391-395`) is deleted — a `Success=false` response means the site *answered*, which is by definition not unreachability. **No Commons message contract changes anywhere in this plan** — the PLAN-08 contract-lock suite (`ClusterClientContractLockTests.cs`) is untouched; the `RouteTo*Response` records are not among the locked seven in any case, and this plan does not modify them. N2/N4/N5 are each one-file classification/decoding/guard fixes with the failing test being exactly the report's scenario.
|
||||
|
||||
**Tech Stack:** C#/.NET 9, xUnit + NSubstitute, ASP.NET Core TestServer (inbound endpoint tests), Roslyn scripting (real compiles in executor tests), Akka.NET (`AskTimeoutException`, transitively referenced via Communication). No EF migrations, no `ManagementActor.cs` edits, no Commons message-contract edits.
|
||||
|
||||
Build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Test per-project: `dotnet test tests/<project>` — targeted filters only, per task.
|
||||
|
||||
## Parallelization (incomplete tasks) — 2026-07-13
|
||||
|
||||
Nothing complete yet. **Ready now** (`blockedBy` empty): **{1, 3, 5, 6}**.
|
||||
|
||||
**Concurrent lanes (dispatch as separate implementers, fully file-disjoint):**
|
||||
- **Inbound consumer lane:** T1 (`ScriptArtifactChangeSubscriber.cs` new + `ServiceCollectionExtensions.cs` + the `Host/Program.cs:95-98` comment) → T2 (docs/tracker truth sweep — no code).
|
||||
- **ESG client lane (single-writer on `ExternalSystemClient.cs` + `ExternalSystemClientTests.cs`):** T3 → T4, serial.
|
||||
- **Inbound routing lane:** T5 (`RouteHelper.cs` + executor/route tests).
|
||||
- **Inbound endpoint lane:** T6 (`EndpointExtensions.cs` + `EndpointContentTypeTests.cs`).
|
||||
|
||||
**Cross-plan file notes:** T1 touches `Host/Program.cs` (comment only, lines 95-98) — coordinate if another active plan holds the `Program.cs` mutex. Nothing here touches `ManagementActor.cs`, `BundleImporter.cs`, or the EF model snapshot.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: `ScriptArtifactChangeSubscriber` — wire the Inbound API as the bus's `ApiMethod` consumer
|
||||
|
||||
**Classification:** standard (new consumer wiring on the hot handler cache; safe-by-design — the per-request revision check remains the correctness fallback, so a missed/duplicate notification is harmless)
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 3, 4, 5, 6
|
||||
**Files:**
|
||||
- Create: `src/ZB.MOM.WW.ScadaBridge.InboundAPI/ScriptArtifactChangeSubscriber.cs`
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.InboundAPI/ServiceCollectionExtensions.cs` (lines 12-31: register the hosted service)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Program.cs` (lines 95-98: rewrite the factually-wrong comment)
|
||||
- Test (create): `tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/ScriptArtifactChangeSubscriberTests.cs`
|
||||
|
||||
The bus contract shipped in PLAN-05 T14 (`IScriptArtifactChangeBus`, `InProcessScriptArtifactChangeBus`, the `BundleImporter` post-commit publisher at `BundleImporter.cs:1811`), and PLAN-06 T3 shipped the `InvalidateMethod` seam — but a repo-wide search confirms **zero `Subscribe` call sites**: the bus publishes into the void while `Host/Program.cs:96-98` claims "the Inbound API compiled-handler cache subscribes as the consumer (wired in plan 06)". This task ships the one real consumer: an `IHostedService` in the InboundAPI project (component-owned, unit-testable — unlike a lambda in the Host composition root). `SharedScript`/`Template` notifications are ignored by this consumer (the executor caches only ApiMethod handlers); their no-consumer status is documented honestly in Task 2.
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Write the failing tests:
|
||||
|
||||
```csharp
|
||||
// tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/ScriptArtifactChangeSubscriberTests.cs
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using NSubstitute;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Entities.InboundApi;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Scripting;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Scripting;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.InboundAPI.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Arch-review R2 N1: the Inbound API must be a REAL subscriber of
|
||||
/// IScriptArtifactChangeBus — a bundle-import publish must invalidate a cached
|
||||
/// handler immediately, without waiting for the per-request revision-check
|
||||
/// self-heal (which remains the correctness fallback per the invalidation
|
||||
/// contract's normative semantics).
|
||||
/// </summary>
|
||||
public class ScriptArtifactChangeSubscriberTests
|
||||
{
|
||||
/// <summary>Minimal in-test bus: records subscriptions, delivers synchronously.</summary>
|
||||
private sealed class RecordingBus : IScriptArtifactChangeBus
|
||||
{
|
||||
private readonly List<Action<ScriptArtifactsChanged>> _handlers = new();
|
||||
public int SubscriberCount => _handlers.Count;
|
||||
|
||||
public void Publish(ScriptArtifactsChanged notification)
|
||||
{
|
||||
foreach (var handler in _handlers.ToArray()) handler(notification);
|
||||
}
|
||||
|
||||
public IDisposable Subscribe(Action<ScriptArtifactsChanged> handler)
|
||||
{
|
||||
_handlers.Add(handler);
|
||||
return new Token(() => _handlers.Remove(handler));
|
||||
}
|
||||
|
||||
private sealed class Token(Action dispose) : IDisposable
|
||||
{
|
||||
public void Dispose() => dispose();
|
||||
}
|
||||
}
|
||||
|
||||
private readonly InboundScriptExecutor _executor = new(
|
||||
NullLogger<InboundScriptExecutor>.Instance, Substitute.For<IServiceProvider>());
|
||||
private readonly RecordingBus _bus = new();
|
||||
private readonly RouteHelper _route = new(
|
||||
Substitute.For<IInstanceLocator>(), Substitute.For<IInstanceRouter>());
|
||||
|
||||
private ScriptArtifactChangeSubscriber CreateSubscriber(IScriptArtifactChangeBus? bus) =>
|
||||
new(_executor, NullLogger<ScriptArtifactChangeSubscriber>.Instance, bus);
|
||||
|
||||
private Task<InboundScriptResult> Run(ApiMethod m) => _executor.ExecuteAsync(
|
||||
m, new Dictionary<string, object?>(), _route, TimeSpan.FromSeconds(10));
|
||||
|
||||
private static ScriptArtifactsChanged ApiMethodChanged(params string[] names) =>
|
||||
new(ScriptArtifactKinds.ApiMethod, names, "BundleImport", DateTimeOffset.UtcNow);
|
||||
|
||||
[Fact]
|
||||
public async Task ApiMethodPublish_InvalidatesCachedHandler_WithoutRevisionCheckSelfHeal()
|
||||
{
|
||||
var subscriber = CreateSubscriber(_bus);
|
||||
await subscriber.StartAsync(CancellationToken.None);
|
||||
|
||||
// Pin a SCRIPT-LESS handler: the RegisterHandler seam is exempt from the
|
||||
// per-request revision check, so the ONLY mechanism that can stop it
|
||||
// serving is an explicit InvalidateMethod — exactly what the bus must trigger.
|
||||
_executor.RegisterHandler("m", async _ => { await Task.CompletedTask; return 99; });
|
||||
var row = new ApiMethod("m", "return 1;") { Id = 1, TimeoutSeconds = 10 };
|
||||
|
||||
var before = await Run(row);
|
||||
Assert.Contains("99", before.ResultJson); // pinned handler serves; no self-heal fires
|
||||
|
||||
_bus.Publish(ApiMethodChanged("m"));
|
||||
|
||||
var after = await Run(row);
|
||||
Assert.True(after.Success);
|
||||
Assert.Contains("1", after.ResultJson); // recompiled from the fresh row — invalidation, not self-heal
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ApiMethodPublish_PurgesKnownBadRecord()
|
||||
{
|
||||
var subscriber = CreateSubscriber(_bus);
|
||||
await subscriber.StartAsync(CancellationToken.None);
|
||||
|
||||
var broken = new ApiMethod("bad", "%%% not C# %%%") { Id = 2, TimeoutSeconds = 10 };
|
||||
Assert.False(_executor.CompileAndRegister(broken));
|
||||
Assert.Equal(1, _executor.KnownBadMethodCount);
|
||||
|
||||
_bus.Publish(ApiMethodChanged("bad"));
|
||||
|
||||
Assert.Equal(0, _executor.KnownBadMethodCount);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task NonApiMethodKinds_AreIgnored()
|
||||
{
|
||||
var subscriber = CreateSubscriber(_bus);
|
||||
await subscriber.StartAsync(CancellationToken.None);
|
||||
|
||||
_executor.RegisterHandler("m", async _ => { await Task.CompletedTask; return 99; });
|
||||
var row = new ApiMethod("m", "return 1;") { Id = 3, TimeoutSeconds = 10 };
|
||||
|
||||
_bus.Publish(new ScriptArtifactsChanged(
|
||||
ScriptArtifactKinds.SharedScript, new[] { "m" }, "BundleImport", DateTimeOffset.UtcNow));
|
||||
_bus.Publish(new ScriptArtifactsChanged(
|
||||
ScriptArtifactKinds.Template, new[] { "m" }, "BundleImport", DateTimeOffset.UtcNow));
|
||||
|
||||
var result = await Run(row);
|
||||
Assert.Contains("99", result.ResultJson); // pinned handler untouched
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task StopAsync_Unsubscribes()
|
||||
{
|
||||
var subscriber = CreateSubscriber(_bus);
|
||||
await subscriber.StartAsync(CancellationToken.None);
|
||||
Assert.Equal(1, _bus.SubscriberCount);
|
||||
|
||||
await subscriber.StopAsync(CancellationToken.None);
|
||||
|
||||
Assert.Equal(0, _bus.SubscriberCount);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task NullBus_NoOps()
|
||||
{
|
||||
// Site roles / plain test hosts have no bus registered — the hosted
|
||||
// service must start and stop cleanly (the revision check alone applies).
|
||||
var subscriber = CreateSubscriber(bus: null);
|
||||
await subscriber.StartAsync(CancellationToken.None);
|
||||
await subscriber.StopAsync(CancellationToken.None);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests --filter ScriptArtifactChangeSubscriberTests` — expect **FAIL** (compile error: no `ScriptArtifactChangeSubscriber` type).
|
||||
|
||||
3. Implement `src/ZB.MOM.WW.ScadaBridge.InboundAPI/ScriptArtifactChangeSubscriber.cs`:
|
||||
|
||||
```csharp
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Scripting;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Scripting;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.InboundAPI;
|
||||
|
||||
/// <summary>
|
||||
/// N1 (arch-review 06 round 2): the Inbound API consumer of the script-artifact
|
||||
/// invalidation contract (docs/plans/2026-07-08-script-artifact-invalidation-contract.md,
|
||||
/// handoff item (a)). Subscribes to <see cref="IScriptArtifactChangeBus"/> on host
|
||||
/// start and, for every <c>ApiMethod</c> notification (e.g. a Transport bundle
|
||||
/// import overwriting method scripts, published post-commit), drops the compiled
|
||||
/// handler AND the known-bad record via
|
||||
/// <see cref="InboundScriptExecutor.InvalidateMethod"/>, so the next request
|
||||
/// recompiles from the fresh row without waiting for the per-request
|
||||
/// revision-check self-heal.
|
||||
/// <para>
|
||||
/// Fast-path only: the revision check in <c>ExecuteAsync</c> remains the
|
||||
/// correctness fallback — this bus is in-process/per-node and at-least-once
|
||||
/// (contract §4), so a missed or duplicate notification must always be harmless.
|
||||
/// The bus is optional (site roles and plain test hosts register none) — the
|
||||
/// subscriber then no-ops. SharedScript/Template notifications are ignored: this
|
||||
/// executor caches only ApiMethod handlers.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public sealed class ScriptArtifactChangeSubscriber : IHostedService
|
||||
{
|
||||
private readonly InboundScriptExecutor _executor;
|
||||
private readonly ILogger<ScriptArtifactChangeSubscriber> _logger;
|
||||
private readonly IScriptArtifactChangeBus? _bus;
|
||||
private IDisposable? _subscription;
|
||||
|
||||
/// <summary>Initializes the subscriber.</summary>
|
||||
/// <param name="executor">The compiled-handler cache to invalidate.</param>
|
||||
/// <param name="logger">Logger instance.</param>
|
||||
/// <param name="bus">The change bus, or null when the host registers none (site roles, tests).</param>
|
||||
public ScriptArtifactChangeSubscriber(
|
||||
InboundScriptExecutor executor,
|
||||
ILogger<ScriptArtifactChangeSubscriber> logger,
|
||||
IScriptArtifactChangeBus? bus = null)
|
||||
{
|
||||
_executor = executor;
|
||||
_logger = logger;
|
||||
_bus = bus;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task StartAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
if (_bus is null)
|
||||
{
|
||||
_logger.LogDebug(
|
||||
"No IScriptArtifactChangeBus registered; inbound handler-cache freshness relies on the per-request revision check alone.");
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
_subscription = _bus.Subscribe(OnChanged);
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task StopAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
_subscription?.Dispose();
|
||||
_subscription = null;
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
private void OnChanged(ScriptArtifactsChanged notification)
|
||||
{
|
||||
if (!string.Equals(
|
||||
notification.ArtifactKind, ScriptArtifactKinds.ApiMethod, StringComparison.Ordinal))
|
||||
{
|
||||
return; // only ApiMethod handlers are cached by this executor
|
||||
}
|
||||
|
||||
foreach (var name in notification.Names)
|
||||
{
|
||||
_executor.InvalidateMethod(name);
|
||||
}
|
||||
|
||||
_logger.LogInformation(
|
||||
"Invalidated {Count} inbound API method handler(s) after a {Source} script-artifact change.",
|
||||
notification.Names.Count, notification.Source);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
In `ServiceCollectionExtensions.AddInboundAPI` (after the `InboundApiEndpointFilter` registration):
|
||||
|
||||
```csharp
|
||||
// N1: the ApiMethod consumer of the script-artifact invalidation bus.
|
||||
// The bus parameter is optional — hosts without a registered
|
||||
// IScriptArtifactChangeBus (site roles, plain test hosts) start it as a no-op
|
||||
// and the executor's per-request revision check alone keeps handlers fresh.
|
||||
services.AddHostedService<ScriptArtifactChangeSubscriber>();
|
||||
```
|
||||
|
||||
In `Host/Program.cs` replace the comment at lines 95-98 with the truth:
|
||||
|
||||
```csharp
|
||||
// Script-artifact invalidation bus: in-process, per-node pub/sub
|
||||
// for ScriptArtifactsChanged. The Transport bundle importer publishes
|
||||
// post-commit; the Inbound API's ScriptArtifactChangeSubscriber (a hosted
|
||||
// service registered by AddInboundAPI below) consumes ApiMethod notifications
|
||||
// and invalidates the compiled-handler cache (wired in plan R2-06 T1; the
|
||||
// per-request revision check remains the correctness fallback).
|
||||
// SharedScript/Template notifications currently have NO consumer — nothing at
|
||||
// central caches compiled artifacts of those kinds. Central-only.
|
||||
```
|
||||
|
||||
4. Run the filter → **PASS**, then the whole project: `dotnet test tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests` → all green (the existing TestServer-based hosts call `AddInboundAPI` + `StartAsync` with no bus registered — the optional-parameter default keeps them green; this is the same optional-dependency pattern `BundleImporter` already uses). Then `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests` → green (Program.cs change is comment-only).
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.InboundAPI/ScriptArtifactChangeSubscriber.cs src/ZB.MOM.WW.ScadaBridge.InboundAPI/ServiceCollectionExtensions.cs src/ZB.MOM.WW.ScadaBridge.Host/Program.cs tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/ScriptArtifactChangeSubscriberTests.cs && git commit -m "fix(inbound): wire the compiled-handler cache as the IScriptArtifactChangeBus ApiMethod consumer (plan R2-06 T1)"`
|
||||
|
||||
### Task 2: Truth sweep — tracker / contract-doc / CLAUDE.md claims about the bus consumer
|
||||
|
||||
**Classification:** small (documentation)
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** 3, 4, 5, 6 (after 1)
|
||||
**Files:**
|
||||
- Modify: `archreview/plans/00-MASTER-TRACKER.md` (line 36 Wave-4 summary: the "`InvalidateMethod` seam consuming PLAN-05's `IScriptArtifactChangeBus` contract (T3)" claim; line 98 cross-plan handoff bullet)
|
||||
- Modify: `docs/plans/2026-07-08-script-artifact-invalidation-contract.md` (line 3 status; §"Handoff to plan 06" rows (a)/(b)/(c), line 58 "no subscribers" sentence)
|
||||
- Modify: `CLAUDE.md` (Templates & Deployment section, the "Plan-06 handoff: … (consumers = plan 06)" sentence)
|
||||
|
||||
N1's second half: the report calls the Host comment (fixed in Task 1) *and* the tracker/doc claims factually wrong — "will misdirect the next person debugging cache coherence." After Task 1 the consumer genuinely exists, so this task makes every claim match reality, including the honest residue: **(b) Management producers are still unwired by design** (the delete path calls `InvalidateMethod` directly at `ManagementActor.cs:2953`; update/edit paths self-heal via the per-request revision check — no notification needed for correctness), and **(c) is satisfied by the revision check** (PLAN-06 T1). SharedScript/Template kinds still publish with no consumer — that is now *documented* as safe (nothing at central caches compiled artifacts of those kinds) instead of silently misdescribed.
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. `00-MASTER-TRACKER.md` line 36: change "`InvalidateMethod` seam consuming PLAN-05's `IScriptArtifactChangeBus` contract (T3)" to "`InvalidateMethod` seam (T3; the bus *subscriber* was NOT wired here — arch-review 06 round-2 N1 — and shipped later via PLAN-R2-06 T1)". Line 98: append to the PLAN-05 T14 ↔ PLAN-06 T3 bullet: "**Closed 2026-07-13 (PLAN-R2-06 T1):** `ScriptArtifactChangeSubscriber` (InboundAPI hosted service) subscribes and invalidates on `ApiMethod` notifications."
|
||||
2. `docs/plans/2026-07-08-script-artifact-invalidation-contract.md`: line 3 status → "Seam + first producer shipped in PLAN-05 (T14). **Consumer (a) shipped in PLAN-R2-06 T1** (`ScriptArtifactChangeSubscriber`). (b) Management producers remain unwired by design — delete calls `InvalidateMethod` directly; edits self-heal via the revision check. (c) is satisfied by the per-request revision check (PLAN-06 T1)." Update the handoff table rows to the same dispositions and rewrite line 58 ("Until (a)–(c) land, plan 05's publisher is a no-op in practice…") to past tense with the shipped/unwired split.
|
||||
3. `CLAUDE.md`: change "(consumers = plan 06)" in the Transport bullet to "(ApiMethod consumer = InboundAPI `ScriptArtifactChangeSubscriber`, wired in plan R2-06; SharedScript/Template publish with no consumer — nothing at central caches those kinds)".
|
||||
4. Verify no stale claim remains: `grep -rn "wired in plan 06\|consumers = plan 06" src docs CLAUDE.md archreview/plans/00-MASTER-TRACKER.md` → zero hits (Task 1 already rewrote the Program.cs comment).
|
||||
5. Commit: `git add archreview/plans/00-MASTER-TRACKER.md docs/plans/2026-07-08-script-artifact-invalidation-contract.md CLAUDE.md && git commit -m "docs(inbound): correct bus-consumer claims in tracker + invalidation-contract doc + CLAUDE.md (plan R2-06 T2)"`
|
||||
|
||||
### Task 3: `DeliverBufferedAsync` parks deterministic `ArgumentException` failures (path template / verb)
|
||||
|
||||
**Classification:** small (classification-only change on the buffered path, mirroring the existing `JsonException` poison fix in the same method)
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** 1, 2, 5, 6
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ExternalSystemClient.cs` (`DeliverBufferedAsync`, lines 214-225: add the catch)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ExternalSystemClientTests.cs` (append)
|
||||
|
||||
N2: `BuildUrl`'s `{param}` substitution throws `ArgumentException` for a placeholder with no matching/non-null parameter (`ExternalSystemClient.cs:535-537`) and `ValidateHttpMethod` throws the same type (`:423-438`). Neither is caught in `DeliverBufferedAsync` (its try at `:214-224` catches only `PermanentExternalSystemException`), and the S&F retry sweep's catch-all treats **any** thrown exception as transient (`StoreAndForwardService.cs:822-830`) — so a method path edited to add `{id}` while calls are buffered burns the full default 50 retries (`StoreAndForwardOptions.cs:21`) with a misleading "transient" `LastError` before parking. Re-running the same substitution against the same stored payload throws deterministically — same poison shape as the `JsonException` fix at `:184-195`. No `StoreAndForwardService` change: the classification decision belongs to the delivery handler, exactly like the existing cases.
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Failing tests (append; reuse the existing `StubResolution`/`BufferedCall`/`MockHttpMessageHandler` helpers — `BufferedCall` builds a payload with `Parameters: null`, which is exactly the unresolvable-placeholder case):
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task DeliverBuffered_PathTemplateNowUnresolvable_ReturnsFalseSoMessageParks()
|
||||
{
|
||||
// Arch-review R2 N2: a method whose path gained a {param} placeholder AFTER
|
||||
// calls were buffered throws ArgumentException deterministically for every
|
||||
// already-buffered message — it must park immediately (permanent), not burn
|
||||
// MaxRetries "transient" attempts (the S&F sweep's catch-all retries any throw).
|
||||
var system = new ExternalSystemDefinition("TestAPI", "https://api.example.com", "none") { Id = 1 };
|
||||
var method = new ExternalSystemMethod("getRecipe", "GET", "/recipes/{id}") { Id = 1, ExternalSystemDefinitionId = 1 };
|
||||
StubResolution(system, method);
|
||||
_httpClientFactory.CreateClient(Arg.Any<string>())
|
||||
.Returns(new HttpClient(new MockHttpMessageHandler(HttpStatusCode.OK, "{}")));
|
||||
var client = new ExternalSystemClient(
|
||||
_httpClientFactory, _repository, NullLogger<ExternalSystemClient>.Instance);
|
||||
|
||||
// BufferedCall carries Parameters:null — the {id} placeholder has no value.
|
||||
var delivered = await client.DeliverBufferedAsync(BufferedCall("TestAPI", "getRecipe"));
|
||||
|
||||
Assert.False(delivered);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task DeliverBuffered_UnsupportedVerbOnStoredMethod_ReturnsFalseSoMessageParks()
|
||||
{
|
||||
// ValidateHttpMethod throws the same deterministic ArgumentException shape.
|
||||
var system = new ExternalSystemDefinition("TestAPI", "https://api.example.com", "none") { Id = 1 };
|
||||
var method = new ExternalSystemMethod("oddVerb", "FOO", "/p") { Id = 1, ExternalSystemDefinitionId = 1 };
|
||||
StubResolution(system, method);
|
||||
var client = new ExternalSystemClient(
|
||||
_httpClientFactory, _repository, NullLogger<ExternalSystemClient>.Instance);
|
||||
|
||||
var delivered = await client.DeliverBufferedAsync(BufferedCall("TestAPI", "oddVerb"));
|
||||
|
||||
Assert.False(delivered);
|
||||
}
|
||||
```
|
||||
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests --filter "DeliverBuffered_PathTemplateNowUnresolvable_ReturnsFalseSoMessageParks|DeliverBuffered_UnsupportedVerbOnStoredMethod_ReturnsFalseSoMessageParks"` → **FAIL** (both tests die on the propagating `ArgumentException` instead of returning `false`).
|
||||
3. Implement — in `DeliverBufferedAsync`, after the `catch (PermanentExternalSystemException …)` branch (line 223), add:
|
||||
|
||||
```csharp
|
||||
catch (ArgumentException ex)
|
||||
{
|
||||
// N2: same poison-message shape as the JsonException catch above. BuildUrl's
|
||||
// {param} path-template substitution (placeholder with no matching/non-null
|
||||
// parameter) and ValidateHttpMethod both throw ArgumentException
|
||||
// deterministically for the SAME stored payload — the method definition
|
||||
// changed underneath the buffered call, and retrying can never succeed.
|
||||
// Return false so the S&F engine parks the message instead of counting the
|
||||
// throw as transient and burning MaxRetries.
|
||||
_logger.LogError(
|
||||
ex,
|
||||
"Buffered call to '{System}'/'{Method}' fails deterministically against the current method definition; parking.",
|
||||
payload.SystemName, payload.MethodName);
|
||||
return false;
|
||||
}
|
||||
```
|
||||
|
||||
Also extend the method's `<summary>` doc-comment (lines 165-173) to name `ArgumentException` alongside malformed JSON as a park-immediately case.
|
||||
4. Run the filter → **PASS**; then the full project `dotnet test tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests` → green; consumer-suite check (no S&F files touched, classification contract only): `dotnet test tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests` → green.
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ExternalSystemClient.cs tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ExternalSystemClientTests.cs && git commit -m "fix(esg): park buffered calls that fail deterministically with ArgumentException (path template/verb) instead of retrying to exhaustion (plan R2-06 T3)"`
|
||||
|
||||
### Task 4: Honor the declared response charset in `ReadBodyBoundedAsync`
|
||||
|
||||
**Classification:** small (decode-only; restores round-1 `ReadAsStringAsync` parity)
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** 1, 2, 5, 6 (NOT 3 — same file; run after 3)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ExternalSystemClient.cs` (`ReadBodyBoundedAsync`, line 482 + new private helper)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ExternalSystemClientTests.cs` (append)
|
||||
|
||||
N4: the bounded read ends with `Encoding.UTF8.GetString(...)` (`:482`) regardless of `Content-Type: …; charset=iso-8859-1` — a legacy external system replying in a non-UTF-8 single-byte encoding becomes mojibake in script-visible response text and error bodies. The pre-P1 `ReadAsStringAsync` honored the charset; restore parity: read `content.Headers.ContentType?.CharSet`, `Encoding.GetEncoding` it (strip optional quotes), fall back to UTF-8 on unknown/invalid values (mojibake beats a hard failure for a cosmetic header).
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Failing tests (append; add the small byte-body handler beside `MockHttpMessageHandler` at line ~1100):
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task Call_Latin1Charset_DecodesUsingDeclaredCharset()
|
||||
{
|
||||
// Arch-review R2 N4: the bounded read must honor the declared charset —
|
||||
// UTF-8-decoding an iso-8859-1 body turns every non-ASCII byte into U+FFFD.
|
||||
var system = new ExternalSystemDefinition("TestAPI", "https://api.example.com", "none") { Id = 1 };
|
||||
var method = new ExternalSystemMethod("getData", "GET", "/data") { Id = 1, ExternalSystemDefinitionId = 1 };
|
||||
StubResolution(system, method);
|
||||
var handler = new ByteBodyHandler(
|
||||
HttpStatusCode.OK,
|
||||
Encoding.Latin1.GetBytes("température 25°C"),
|
||||
"text/plain; charset=iso-8859-1");
|
||||
_httpClientFactory.CreateClient(Arg.Any<string>()).Returns(new HttpClient(handler));
|
||||
var client = new ExternalSystemClient(
|
||||
_httpClientFactory, _repository, NullLogger<ExternalSystemClient>.Instance);
|
||||
|
||||
var result = await client.CallAsync("TestAPI", "getData");
|
||||
|
||||
Assert.True(result.Success);
|
||||
Assert.Contains("température 25°C", result.ResponseJson);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Call_UnknownCharset_FallsBackToUtf8()
|
||||
{
|
||||
var system = new ExternalSystemDefinition("TestAPI", "https://api.example.com", "none") { Id = 1 };
|
||||
var method = new ExternalSystemMethod("getData", "GET", "/data") { Id = 1, ExternalSystemDefinitionId = 1 };
|
||||
StubResolution(system, method);
|
||||
var handler = new ByteBodyHandler(
|
||||
HttpStatusCode.OK, Encoding.UTF8.GetBytes("ok"), "text/plain; charset=not-a-charset");
|
||||
_httpClientFactory.CreateClient(Arg.Any<string>()).Returns(new HttpClient(handler));
|
||||
var client = new ExternalSystemClient(
|
||||
_httpClientFactory, _repository, NullLogger<ExternalSystemClient>.Instance);
|
||||
|
||||
var result = await client.CallAsync("TestAPI", "getData");
|
||||
|
||||
Assert.True(result.Success);
|
||||
Assert.Contains("ok", result.ResponseJson);
|
||||
}
|
||||
|
||||
/// <summary>Test helper: returns a fixed byte body with an explicit Content-Type.</summary>
|
||||
private sealed class ByteBodyHandler(
|
||||
HttpStatusCode statusCode, byte[] body, string contentType) : HttpMessageHandler
|
||||
{
|
||||
protected override Task<HttpResponseMessage> SendAsync(
|
||||
HttpRequestMessage request, CancellationToken cancellationToken)
|
||||
{
|
||||
var response = new HttpResponseMessage(statusCode) { Content = new ByteArrayContent(body) };
|
||||
response.Content.Headers.ContentType = MediaTypeHeaderValue.Parse(contentType);
|
||||
return Task.FromResult(response);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests --filter "Call_Latin1Charset_DecodesUsingDeclaredCharset|Call_UnknownCharset_FallsBackToUtf8"` → **FAIL** (`Call_Latin1Charset…`: UTF-8 decode yields U+FFFD for `é`/`°`; the fallback test passes already — it pins the tolerance so the fix cannot regress it into a throw).
|
||||
3. Implement — replace line 482 with:
|
||||
|
||||
```csharp
|
||||
// N4: honor the response's declared charset (parity with the pre-bounded-read
|
||||
// ReadAsStringAsync). Unknown/invalid charset → UTF-8 fallback: mojibake beats
|
||||
// failing the call over a cosmetic header.
|
||||
return ResolveEncoding(content.Headers.ContentType?.CharSet)
|
||||
.GetString(buffer.GetBuffer(), 0, (int)buffer.Length);
|
||||
```
|
||||
|
||||
and add below `ReadBodyBoundedAsync`:
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Resolves the declared response charset to an <see cref="Encoding"/>, tolerating
|
||||
/// the quoted form some servers emit (<c>charset="iso-8859-1"</c>). Null, empty,
|
||||
/// or unrecognized values fall back to UTF-8.
|
||||
/// </summary>
|
||||
private static Encoding ResolveEncoding(string? charset)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(charset))
|
||||
{
|
||||
return Encoding.UTF8;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
return Encoding.GetEncoding(charset.Trim().Trim('"'));
|
||||
}
|
||||
catch (ArgumentException)
|
||||
{
|
||||
return Encoding.UTF8;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Also update the `ReadBodyBoundedAsync` doc-comment (lines 440-450) to state the charset behavior.
|
||||
4. Run the filter → **PASS**; full project run `dotnet test tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests` → green (the oversized-body and timeout tests at `:1047` are decode-independent).
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ExternalSystemClient.cs tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ExternalSystemClientTests.cs && git commit -m "fix(esg): honor the declared response charset in ReadBodyBoundedAsync with UTF-8 fallback (plan R2-06 T4)"`
|
||||
|
||||
### Task 5: Typed `SITE_UNREACHABLE` — `AskTimeoutException` classification replaces message sniffing
|
||||
|
||||
**Classification:** standard (reclassifies the routed-failure taxonomy for every inbound routed call; the change also makes SITE_UNREACHABLE fire for the *real* unreachable path for the first time)
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1, 2, 3, 4, 6
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.InboundAPI/RouteHelper.cs` (lines 16-21: inner-exception ctor; 185, 233, 297, 348: the four router awaits; 364-377: `ResolveSiteAsync` comment; 379-395: `RoutingFailure` + delete `IsUnreachable`)
|
||||
- Modify: `docs/requirements/Component-InboundAPI.md` (error-code table: `SITE_UNREACHABLE` = "the site did not respond to the routed request (no contact / down / partitioned)")
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/InboundScriptExecutorTests.cs` (rewrite the sniff-pinning test at lines 100-124, add the anti-sniff test), `tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/RouteHelperTests.cs` (append)
|
||||
|
||||
N3: `RouteHelper.IsUnreachable` (`:391-395`) matches `"unreachable"`/`"no contact"` substrings in `ErrorMessage` text produced by another component — a harmless rewording silently downgrades `SITE_UNREACHABLE` to `SCRIPT_ERROR` with no test failing at the boundary. The typed signal already exists in the communication layer: an unreachable site means the ClusterClient Ask **expires** (`CentralCommunicationActor.HandleSiteEnvelope` deliberately drops enveloped messages for sites with no client so "the Ask will timeout on the caller side"; `CommunicationService.RouteTo*Async` are plain bounded Asks), surfacing as `Akka.Actor.AskTimeoutException` — which today falls into the executor's generic catch as `SCRIPT_ERROR`. Conversely, a `Success=false` *response* means the site **answered** — by definition not unreachability. So: type `AskTimeoutException → SiteUnreachableException` at the four router awaits, and make `RoutingFailure` always a plain `InvalidOperationException`. **No Commons message-contract change** (the typed-exception form was chosen over an additive error-code field on the `RouteTo*Response` records, so the PLAN-08 contract-lock suite needs no update; those records are not among the locked seven anyway, and additive-only + default(T)-safe rules would apply if a future change adds such a field).
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Failing tests. In `InboundScriptExecutorTests.cs`, **rewrite** `RoutedSiteUnreachable_Returns500WithSiteUnreachableCode` (lines 100-124 — it currently pins the sniff by stubbing `ErrorMessage: "Site unreachable"`) and add the anti-sniff case:
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task RoutedAskTimeout_Returns500WithSiteUnreachableCode()
|
||||
{
|
||||
// N3: unreachability is the Ask EXPIRING (site never answered) — typed as
|
||||
// AskTimeoutException by the communication layer — not a substring in a
|
||||
// failure message. Pre-fix this fell into the generic catch as SCRIPT_ERROR.
|
||||
var locator = Substitute.For<IInstanceLocator>();
|
||||
locator.GetSiteIdForInstanceAsync("X", Arg.Any<CancellationToken>()).Returns("SiteA");
|
||||
var router = Substitute.For<IInstanceRouter>();
|
||||
router.RouteToCallAsync("SiteA", Arg.Any<RouteToCallRequest>(), Arg.Any<CancellationToken>())
|
||||
.Returns(Task.FromException<RouteToCallResponse>(
|
||||
new Akka.Actor.AskTimeoutException("Timeout after 00:00:30")));
|
||||
var route = new RouteHelper(locator, router);
|
||||
|
||||
var method = new ApiMethod("routed", "return await Route.To(\"X\").Call(\"s\");")
|
||||
{ Id = 7, TimeoutSeconds = 10 };
|
||||
|
||||
var result = await _executor.ExecuteAsync(
|
||||
method, new Dictionary<string, object?>(), route, TimeSpan.FromSeconds(10));
|
||||
|
||||
Assert.False(result.Success);
|
||||
Assert.Equal("SITE_UNREACHABLE", result.ErrorCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SiteRespondedFailure_MessageMentioningUnreachable_IsScriptErrorNotSiteUnreachable()
|
||||
{
|
||||
// N3 anti-sniff: a site that RESPONDED with Success=false is reachable by
|
||||
// definition — even when its error text happens to contain "unreachable".
|
||||
// Pre-fix the substring sniff misclassified this as SITE_UNREACHABLE.
|
||||
var locator = Substitute.For<IInstanceLocator>();
|
||||
locator.GetSiteIdForInstanceAsync("X", Arg.Any<CancellationToken>()).Returns("SiteA");
|
||||
var router = Substitute.For<IInstanceRouter>();
|
||||
router.RouteToCallAsync("SiteA", Arg.Any<RouteToCallRequest>(), Arg.Any<CancellationToken>())
|
||||
.Returns(ci => new RouteToCallResponse(
|
||||
((RouteToCallRequest)ci[1]).CorrelationId,
|
||||
Success: false, ReturnValue: null,
|
||||
ErrorMessage: "PLC device unreachable behind the site gateway",
|
||||
DateTimeOffset.UtcNow));
|
||||
var route = new RouteHelper(locator, router);
|
||||
|
||||
var method = new ApiMethod("routed2", "return await Route.To(\"X\").Call(\"s\");")
|
||||
{ Id = 8, TimeoutSeconds = 10 };
|
||||
|
||||
var result = await _executor.ExecuteAsync(
|
||||
method, new Dictionary<string, object?>(), route, TimeSpan.FromSeconds(10));
|
||||
|
||||
Assert.False(result.Success);
|
||||
Assert.Equal("SCRIPT_ERROR", result.ErrorCode);
|
||||
}
|
||||
```
|
||||
|
||||
In `RouteHelperTests.cs` (uses the existing `_locator`/`_router`/`CreateHelper`/`SiteResolves` scaffolding), append:
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task Call_AskTimeout_ThrowsSiteUnreachableException()
|
||||
{
|
||||
SiteResolves("inst", "SiteA");
|
||||
_router.RouteToCallAsync("SiteA", Arg.Any<RouteToCallRequest>(), Arg.Any<CancellationToken>())
|
||||
.Returns(Task.FromException<RouteToCallResponse>(
|
||||
new Akka.Actor.AskTimeoutException("Timeout after 00:00:30")));
|
||||
|
||||
await Assert.ThrowsAsync<SiteUnreachableException>(
|
||||
() => CreateHelper().To("inst").Call("s"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task GetAttributes_AskTimeout_ThrowsSiteUnreachableException()
|
||||
{
|
||||
SiteResolves("inst", "SiteA");
|
||||
_router.RouteToGetAttributesAsync("SiteA", Arg.Any<RouteToGetAttributesRequest>(), Arg.Any<CancellationToken>())
|
||||
.Returns(Task.FromException<RouteToGetAttributesResponse>(
|
||||
new Akka.Actor.AskTimeoutException("Timeout after 00:00:30")));
|
||||
|
||||
await Assert.ThrowsAsync<SiteUnreachableException>(
|
||||
() => CreateHelper().To("inst").GetAttributes(new[] { "a" }));
|
||||
}
|
||||
```
|
||||
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests --filter "RoutedAskTimeout_Returns500WithSiteUnreachableCode|SiteRespondedFailure_MessageMentioningUnreachable|AskTimeout_ThrowsSiteUnreachableException"` → **FAIL** (Ask-timeout cases surface `SCRIPT_ERROR`/`AskTimeoutException`; the anti-sniff case returns `SITE_UNREACHABLE`).
|
||||
3. Implement in `RouteHelper.cs` (add `using Akka.Actor;`):
|
||||
- `SiteUnreachableException` gains `public SiteUnreachableException(string message, Exception innerException) : base(message, innerException) { }` and its class doc-comment is rewritten: thrown when the routed Ask **expires** (typed `AskTimeoutException` from the communication layer) — never inferred from message text.
|
||||
- Add to `RouteTarget`:
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// N3: awaits a routed Ask and TYPES unreachability. AskTimeoutException means
|
||||
/// the site never answered (no ClusterClient contact, site down, partition) —
|
||||
/// the communication layer's own typed signal (CentralCommunicationActor drops
|
||||
/// envelopes for contactless sites so the Ask expires caller-side) — and
|
||||
/// surfaces as SiteUnreachableException so the executor maps it to the spec's
|
||||
/// SITE_UNREACHABLE code. A Success=false RESPONSE means the site answered and
|
||||
/// is therefore reachable; it flows through RoutingFailure as a plain
|
||||
/// InvalidOperationException (SCRIPT_ERROR).
|
||||
/// </summary>
|
||||
private static async Task<TResponse> RouteOrUnreachable<TResponse>(Task<TResponse> routedAsk)
|
||||
{
|
||||
try
|
||||
{
|
||||
return await routedAsk;
|
||||
}
|
||||
catch (AskTimeoutException ex)
|
||||
{
|
||||
throw new SiteUnreachableException(
|
||||
"Site did not respond to the routed request within the routing timeout — unreachable or not connected.",
|
||||
ex);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- Wrap the four router awaits, e.g. line 185: `var response = await RouteOrUnreachable(_instanceRouter.RouteToCallAsync(siteId, request, token));` (same at 233, 297, 348).
|
||||
- Replace `RoutingFailure` + delete `IsUnreachable` (lines 379-395):
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// C3/N3: a routing-level failure REPORTED BY the site or the locator. The
|
||||
/// remote end answered (or the instance simply has no site), so this is never
|
||||
/// unreachability — always a plain <see cref="InvalidOperationException"/>
|
||||
/// (SCRIPT_ERROR at the endpoint). Unreachability is typed at the Ask boundary
|
||||
/// by <c>RouteOrUnreachable</c> (AskTimeoutException → SiteUnreachableException),
|
||||
/// replacing the old message-substring sniff that a harmless rewording in the
|
||||
/// communication layer could silently defeat.
|
||||
/// </summary>
|
||||
private static InvalidOperationException RoutingFailure(string message) => new(message);
|
||||
```
|
||||
|
||||
- Fix the now-stale comment inside `ResolveSiteAsync` (lines 368-371) — a null site id is a plain not-found, never sniffed.
|
||||
- Update the `Component-InboundAPI.md` error-code table row for `SITE_UNREACHABLE` to the typed semantics ("the target site did not respond to the routed request within the routing timeout").
|
||||
4. Run the filter → **PASS**; full project `dotnet test tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests` → green (no other test stubs unreachable-flavored `ErrorMessage` text).
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.InboundAPI/RouteHelper.cs docs/requirements/Component-InboundAPI.md tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/InboundScriptExecutorTests.cs tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/RouteHelperTests.cs && git commit -m "fix(inbound): type SITE_UNREACHABLE on AskTimeoutException at the router await instead of message-substring sniffing (plan R2-06 T5)"`
|
||||
|
||||
### Task 6: 415 guard covers chunked (no Content-Length) bodies
|
||||
|
||||
**Classification:** small (guard-condition fix on the endpoint; also fixes lenient parsing of chunked bodies)
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** 1, 2, 3, 4, 5
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.InboundAPI/EndpointExtensions.cs` (lines 182-212: the 415 guard at 187-194 and the parse condition at 206-207)
|
||||
- Modify: `docs/requirements/Component-InboundAPI.md` (the 415 row: "a body — declared by Content-Length or Transfer-Encoding (chunked)")
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/EndpointContentTypeTests.cs` (append)
|
||||
|
||||
N5: the 415 guard keys on `ContentLength > 0` (`:187`); a chunked non-JSON body has null `ContentLength`, skips the 415, then also skips JSON parsing (the parse condition `:206-207` is `ContentLength > 0 || ContentType contains json`), so `body = null` and the caller gets a misleading `VALIDATION_FAILED` — the same error class S9 set out to fix, now confined to the chunked edge. Fix: compute `hasBody` once (Content-Length positive, OR Content-Length absent with a `Transfer-Encoding` header present) and use it in **both** the 415 guard and the parse condition — so a chunked JSON/no-Content-Type body also parses leniently, matching the fixed-length behavior. The round-1 leniencies are preserved: no body → no 415; body with no Content-Type → lenient JSON parse.
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Failing tests (append to `EndpointContentTypeTests`, reusing `BuildHostAsync`/`SeedKeyAsync`; add the non-seekable content helper so the client can never auto-compute a Content-Length):
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task ChunkedNonJsonBody_Returns415()
|
||||
{
|
||||
// Arch-review R2 N5: a chunked body has NULL ContentLength — it must not
|
||||
// slip past the 415 guard into a misleading VALIDATION_FAILED.
|
||||
const string methodName = "echoChunked";
|
||||
var method = new ApiMethod(methodName, "return Parameters[\"value\"];")
|
||||
{
|
||||
Id = 1,
|
||||
TimeoutSeconds = 10,
|
||||
ParameterDefinitions = """[{"name":"value","type":"Integer","required":true}]""",
|
||||
};
|
||||
var repo = Substitute.For<IInboundApiRepository>();
|
||||
repo.GetMethodByNameAsync(methodName, Arg.Any<CancellationToken>()).Returns(method);
|
||||
|
||||
using var host = await BuildHostAsync(repo);
|
||||
var token = await SeedKeyAsync(host, methodName);
|
||||
var client = host.GetTestClient();
|
||||
|
||||
var request = new HttpRequestMessage(HttpMethod.Post, "/api/" + methodName)
|
||||
{
|
||||
Content = new ChunkedByteContent(Encoding.UTF8.GetBytes("hello")),
|
||||
};
|
||||
request.Content.Headers.ContentType = MediaTypeHeaderValue.Parse("text/plain");
|
||||
request.Headers.TransferEncodingChunked = true;
|
||||
request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", token);
|
||||
|
||||
var response = await client.SendAsync(request);
|
||||
var body = await response.Content.ReadAsStringAsync();
|
||||
|
||||
Assert.Equal(HttpStatusCode.UnsupportedMediaType, response.StatusCode);
|
||||
Assert.Contains("UNSUPPORTED_MEDIA_TYPE", body);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ChunkedJsonBody_NoContentType_StillParsesLeniently()
|
||||
{
|
||||
// The parse condition must also learn about chunked bodies: a chunked JSON
|
||||
// body with no Content-Type previously skipped parsing entirely (body=null →
|
||||
// misleading "missing required parameter").
|
||||
const string methodName = "echoChunkedJson";
|
||||
var method = new ApiMethod(methodName, "return Parameters[\"value\"];")
|
||||
{
|
||||
Id = 1,
|
||||
TimeoutSeconds = 10,
|
||||
ParameterDefinitions = """[{"name":"value","type":"Integer","required":true}]""",
|
||||
};
|
||||
var repo = Substitute.For<IInboundApiRepository>();
|
||||
repo.GetMethodByNameAsync(methodName, Arg.Any<CancellationToken>()).Returns(method);
|
||||
|
||||
using var host = await BuildHostAsync(repo);
|
||||
var token = await SeedKeyAsync(host, methodName);
|
||||
var client = host.GetTestClient();
|
||||
|
||||
var request = new HttpRequestMessage(HttpMethod.Post, "/api/" + methodName)
|
||||
{
|
||||
Content = new ChunkedByteContent(Encoding.UTF8.GetBytes("{\"value\":42}")),
|
||||
};
|
||||
request.Content.Headers.ContentType = null;
|
||||
request.Headers.TransferEncodingChunked = true;
|
||||
request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", token);
|
||||
|
||||
var response = await client.SendAsync(request);
|
||||
var body = await response.Content.ReadAsStringAsync();
|
||||
|
||||
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
|
||||
Assert.Contains("42", body);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Content whose length is never computable, forcing HttpClient to transmit it
|
||||
/// chunked (no Content-Length header ever reaches the server).
|
||||
/// </summary>
|
||||
private sealed class ChunkedByteContent(byte[] bytes) : HttpContent
|
||||
{
|
||||
protected override Task SerializeToStreamAsync(
|
||||
Stream stream, System.Net.TransportContext? context) =>
|
||||
stream.WriteAsync(bytes, 0, bytes.Length);
|
||||
|
||||
protected override bool TryComputeLength(out long length)
|
||||
{
|
||||
length = -1;
|
||||
return false;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests --filter "ChunkedNonJsonBody_Returns415|ChunkedJsonBody_NoContentType_StillParsesLeniently"` → **FAIL** (first: 400 `VALIDATION_FAILED` instead of 415; second: 400 "missing required parameter" instead of 200).
|
||||
3. Implement in `EndpointExtensions.cs` — above the S9 block (line ~182), compute the body signal once, then use it in both places:
|
||||
|
||||
```csharp
|
||||
// N5: a body is present when Content-Length says so, OR when the request is
|
||||
// chunked — Transfer-Encoding present and Content-Length (necessarily) absent.
|
||||
// Computed once and used by BOTH the 415 guard and the JSON parse condition so
|
||||
// chunked and fixed-length bodies behave identically.
|
||||
var requestHeaders = httpContext.Request.Headers;
|
||||
var hasBody = httpContext.Request.ContentLength > 0
|
||||
|| (httpContext.Request.ContentLength is null && requestHeaders.TransferEncoding.Count > 0);
|
||||
```
|
||||
|
||||
The 415 guard (lines 187-189) becomes `if (hasBody && !string.IsNullOrEmpty(httpContext.Request.ContentType) && httpContext.Request.ContentType.Contains("json", StringComparison.OrdinalIgnoreCase) == false)`; the parse condition (lines 206-207) becomes `if (hasBody || httpContext.Request.ContentType?.Contains("json", StringComparison.OrdinalIgnoreCase) == true)`. Update the S9 comment block (lines 182-186) to name the chunked case. Update the doc's 415 row.
|
||||
4. Run the filter → **PASS**; full project `dotnet test tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests` → green (`NonJsonContentType_WithBody_Returns415` and the lenient no-Content-Type case at `EndpointContentTypeTests.cs:96-146` still pass — fixed-length behavior is unchanged, and a zero-length body still never 415s).
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.InboundAPI/EndpointExtensions.cs docs/requirements/Component-InboundAPI.md tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/EndpointContentTypeTests.cs && git commit -m "fix(inbound): 415 guard and lenient JSON parse cover chunked (no Content-Length) bodies (plan R2-06 T6)"`
|
||||
|
||||
---
|
||||
|
||||
## Dependencies on other plans
|
||||
|
||||
- **PLAN-R2-05 owns the publisher-side Add-skip fix.** `BundleImporter.PublishScriptArtifactChanges` (`BundleImporter.cs:1777-1817`, publish at `:1811`) deliberately excludes `Add` resolutions ("nothing is cached under a brand-new name yet"); revisiting that rationale is PLAN-R2-05's finding. Task 1's subscriber has **no `blockedBy`** on it — the bus contract shipped in PLAN-05 T14, the subscriber consumes whatever is published, and `InvalidateMethod` is safe for unknown names, so when PLAN-R2-05 broadens the publisher the consumer needs zero changes. Files are disjoint (`BundleImporter.cs` vs `InboundAPI/*`) — safe to run both plans concurrently.
|
||||
- **Contract-lock (PLAN-08 T9).** This plan touches no Commons message contracts, so `ClusterClientContractLockTests.cs` (locked records: `CreateDataConnectionCommand`, `DeployInstanceCommand`, `NotificationSubmit`, `AttributeValueChanged`, `GetAttributeRequest`, `SetStaticAttributeCommand`, `ManagementEnvelope`) needs no update. Task 5 deliberately chose the typed-exception form over an additive error-code field on the `RouteTo*Response` records; those records are not among the locked seven, but any future additive field there must still be safe at `default(T)` (the locked reality: Akka's Newtonsoft serializer ignores C# optional-parameter defaults on missing fields).
|
||||
- **`Host/Program.cs` mutex.** Task 1's edit is comment-only (lines 95-98) — coordinate only if another active plan is editing `Program.cs` in the same window.
|
||||
|
||||
## Execution order
|
||||
|
||||
**P0 (start immediately, independent, file-disjoint):** Tasks 1, 3, 5, 6 in parallel lanes.
|
||||
|
||||
- **Inbound consumer lane:** 1 → 2 (docs truth sweep last, so it records what actually shipped).
|
||||
- **ESG client lane:** 3 → 4 (both edit `ExternalSystemClient.cs` + the same test file — serial).
|
||||
- **Inbound routing lane:** 5 (free-floating).
|
||||
- **Inbound endpoint lane:** 6 (free-floating).
|
||||
|
||||
Intra-plan critical path: **3 → 4** and **1 → 2** (both two deep); everything else is single-task. No EF migrations, no `ManagementActor.cs`, no initiative-wide mutexes beyond the `Program.cs` comment note above.
|
||||
|
||||
## Findings Coverage
|
||||
|
||||
| # | Finding (report section) | Severity | Task(s) |
|
||||
|---|--------------------------|----------|---------|
|
||||
| N1 | Script-artifact invalidation bus publishes into the void; Host comment + master tracker claim a consumer that does not exist (the open PLAN-05→PLAN-06 handoff) | Medium | 1 (subscriber + Program.cs comment), 2 (tracker/contract-doc/CLAUDE.md truth sweep) |
|
||||
| N2 | Now-unresolvable path template (or bad verb) on a buffered message classified transient and retried to exhaustion (50 retries) before parking with a misleading `LastError` | Low | 3 |
|
||||
| N3 | `SITE_UNREACHABLE` classification is substring-sniffing on another component's error-message text; a rewording silently downgrades it to `SCRIPT_ERROR` | Low | 5 (typed `AskTimeoutException → SiteUnreachableException` at the router await; sniff deleted; no Commons contract change → contract-lock suite untouched) |
|
||||
| N4 | Bounded response read decodes UTF-8 unconditionally, dropping the round-1 charset honoring (mojibake for non-UTF-8 legacy systems) | Low | 4 |
|
||||
| N5 | 415 guard keyed on `ContentLength > 0` is bypassed by chunked non-JSON bodies, resurfacing the misleading-error class S9 fixed | Low | 6 |
|
||||
| U2 (round-1 partial) | No cache-coherence mechanism for compiled handlers — the revision check shipped but the claimed bus consumer was never wired | Underdeveloped | Same as N1 → 1, 2 (the report folds the U2 residue into N1) |
|
||||
| U4 (round-1 partial) | Twilio accept-only: status-callback webhook / per-recipient delivery state | Underdeveloped | **Deferred (unchanged)** — stays out of scope per the round-1 ruling (documented in `Component-NotificationService.md:95`): an inbound webhook endpoint is new attack surface warranting its own design pass. The residual minor (SID rows saved before validation are not retro-checked) is accepted. No task. |
|
||||
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"planPath": "archreview/plans/PLAN-R2-06-edge-integrations.md",
|
||||
"tasks": [
|
||||
{ "id": 1, "subject": "Task 1: ScriptArtifactChangeSubscriber — wire the Inbound API as the bus's ApiMethod consumer", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 2, "subject": "Task 2: Truth sweep — tracker / contract-doc / CLAUDE.md claims about the bus consumer", "status": "pending", "blockedBy": [1] },
|
||||
{ "id": 3, "subject": "Task 3: DeliverBufferedAsync parks deterministic ArgumentException failures (path template / verb)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 4, "subject": "Task 4: Honor the declared response charset in ReadBodyBoundedAsync", "status": "pending", "blockedBy": [3] },
|
||||
{ "id": 5, "subject": "Task 5: Typed SITE_UNREACHABLE — AskTimeoutException classification replaces message sniffing", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 6, "subject": "Task 6: 415 guard covers chunked (no Content-Length) bodies", "status": "pending", "blockedBy": [] }
|
||||
],
|
||||
"lastUpdated": "2026-07-13T03:23:07Z"
|
||||
}
|
||||
@@ -0,0 +1,980 @@
|
||||
# PLAN-R2-07 — UI, Management & Security Round-2 Hardening Implementation Plan
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
|
||||
|
||||
**Goal** — Close every NEW (round-2) finding from architecture review 07 round 2 (`archreview/07-ui-management-security.md`, dated 2026-07-12): throttle the last unthrottled LDAP-bind surface (the debug-stream hub) and fix the doc that falsely claims coverage is complete (N1); make `LoginThrottle` keys survive the Traefik topology via trusted-proxy `ForwardedHeaders` and document the username-lockout-DoS trade-off (N2); enforce LDAP-mapping site scope on all four secured-write handlers, where the spec already promises it (N3); stop the Alarm Summary poll from displaying cross-site data after a site switch (N4) and from regressing fresher live state (N5); un-stick `IsLive` when a live-cache aggregator dies (N6); adopt the house disposal guard on the live callback (N7); amortize `LoginThrottle.Prune` (N8); and freeze the secret-scrubber's fragment coverage with a lock-in test (N9). Round-1 partials with documented deferrals (S1 import idempotency, S4 streaming multipart, P2 QueryDeployments/ExportBundle internals, UA6 fleet-wide grid a11y) **stay deferred** — coverage rows only, no tasks.
|
||||
|
||||
**Architecture** — All fixes stay inside the seams round 1 built: `ManagementAuthenticator` is already the shared Basic→LDAP+throttle flow, so N1 is *deleting* the hub's bespoke bind and delegating; `LoginThrottle` and `SecurityOptions` already live in the Security component, so N2 adds one small pure setup class (`ForwardedHeadersSetup`) wired at the Host composition root ahead of the middleware pipeline; N3 reuses the existing `EnforceSiteScope`/`EnforceSiteScopeForIdentifier` helpers (`ManagementActor.cs:495-536`) exactly as C2's fix did for `DeployArtifacts`; N4/N5/N7 mirror guards that already exist elsewhere in the same file or in `DebugView.razor`; N6 adds a deathwatch hook to the (already reference-counted, lock-guarded) `SiteAlarmLiveCacheService`. Message-contract and repository changes are strictly additive (optional trailing params). Every spec-affecting change updates `docs/requirements/Component-Security.md` / `Component-CentralUI.md` / `Component-Communication.md` in the same task as the code.
|
||||
|
||||
**Tech Stack** — C#/.NET 10, Akka.NET (TestKit.Xunit2 + NSubstitute for actor tests), ASP.NET Core minimal endpoints + SignalR + Blazor Server (bUnit), EF Core (ConfigurationDatabase). Build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Test per-project: `dotnet test tests/<project>` (CLI.Tests is in the slnx since PLAN-08 T1 — no special handling).
|
||||
|
||||
---
|
||||
|
||||
## Parallelization & mutex rules
|
||||
|
||||
**ManagementActor single-writer lane (initiative-wide rule, same as PLAN-07):** every task that edits `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs` serializes into ONE lane — among themselves **and** against any other plan's ManagementActor task. In this round-2 wave only PLAN-R2-07 touches the file (**Tasks 4 and 6** — run T4 → T6, never concurrently), but the rule stands: check the round-2 master tracker before starting either, and if another plan later gains a ManagementActor task, this lane serializes with it.
|
||||
|
||||
**`SiteAlarmLiveCacheService.cs` mutex with PLAN-R2-02:** PLAN-R2-02 also modifies `src/ZB.MOM.WW.ScadaBridge.Communication/SiteAlarmLiveCacheService.cs` (delta coalescing). **Task 10 (N6) and R2-02's coalescing task must NOT run concurrently** — whichever plan's task starts first finishes and lands its commit before the other begins, and the second task rebases on the first's committed state. Coordinate through the round-2 tracker.
|
||||
|
||||
**`AlarmSummary.razor` internal lane:** Tasks 7, 8, 9 all edit the same file/methods — run strictly 7 → 8 → 9.
|
||||
|
||||
**Concurrent lanes at plan start:**
|
||||
- **Free / file-disjoint:** T1 (`DebugStreamHub.cs`), T2 (`Security/ForwardedHeadersSetup.cs` + `Host/Program.cs`), T5 (`ISecuredWriteRepository` + EF repo), T7 (first of the AlarmSummary lane), T10 (subject to the R2-02 mutex), T11 (`LoginThrottle.cs`), T12 (test-only + `ConfigSecretScrubber.cs` xmldoc).
|
||||
- **Ordering constraints:** 3←2; 6←(4,5); 8←7; 9←8. T4 and T6 share the ManagementActor lane (T4 first).
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Route the DebugStreamHub LDAP bind through `ManagementAuthenticator` (throttled) + fix the false doc claim
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 2, 4, 5, 7, 10, 11, 12
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/DebugStreamHub.cs` (lines 78–146: replace the bespoke bypass + Basic-decode + LDAP-bind + role-map block in `OnConnectedAsync`)
|
||||
- Modify: `docs/requirements/Component-Security.md` (line 208, "Scope — every LDAP-bind surface" bullet)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/DebugStreamHubTests.cs` (extend — currently only tests the static `IsInstanceAccessAllowed`)
|
||||
|
||||
The hub currently uses `ManagementAuthenticator` **only** for the DisableLogin bypass (`TryDisableLoginUser`, :82) and then runs its own Basic decode + `ILdapAuthService.AuthenticateAsync` (:119-120) + `RoleMapper` (:129-130) — so the `LoginThrottle` never sees hub connection attempts, and hub failures don't feed the shared lockout. `ManagementAuthenticator.AuthenticateAsync` (`ManagementAuthenticator.cs:70-146`) already does all of it — bypass, decode, `IsLockedOut` refusal, bind, `RecordFailure`/`RecordSuccess`, role resolution, and system-wide `PermittedSiteIds` normalization. Delegate to it.
|
||||
|
||||
1. Write the failing tests. Harness: a real `DefaultHttpContext` whose `RequestServices` carries `IOptions<AuthDisableLoginOptions>` (off), `IOptions<SecurityOptions>` (defaults), a `FakeTimeProvider`-style clock as `TimeProvider`, a singleton `LoginThrottle`, and a substituted `ILdapAuthService` — mirror `ManagementAuthenticatorTests.BasicAuthContext` exactly. Wrap it for SignalR with a minimal `HubCallerContext` fake:
|
||||
```csharp
|
||||
private sealed class TestHubCallerContext : HubCallerContext
|
||||
{
|
||||
private readonly FeatureCollection _features = new();
|
||||
public TestHubCallerContext(HttpContext http) =>
|
||||
_features.Set<IHttpContextFeature>(new TestHttpContextFeature { HttpContext = http });
|
||||
public bool Aborted { get; private set; }
|
||||
public override string ConnectionId => "test-conn";
|
||||
public override string? UserIdentifier => null;
|
||||
public override System.Security.Claims.ClaimsPrincipal? User => null;
|
||||
public override IDictionary<object, object?> Items { get; } = new Dictionary<object, object?>();
|
||||
public override IFeatureCollection Features => _features;
|
||||
public override CancellationToken ConnectionAborted => CancellationToken.None;
|
||||
public override void Abort() => Aborted = true;
|
||||
|
||||
private sealed class TestHttpContextFeature : IHttpContextFeature { public HttpContext? HttpContext { get; set; } }
|
||||
}
|
||||
```
|
||||
Build the hub with real light-weight collaborators (`DebugStreamService` needs `CommunicationService` — construct it directly: `new CommunicationService(Options.Create(new CommunicationOptions()), NullLogger<CommunicationService>.Instance)` — plus a `new SiteStreamGrpcClientFactory(NullLoggerFactory.Instance)` and a bare `ServiceCollection().BuildServiceProvider()`; `IHubContext<DebugStreamHub>` is `Substitute.For`), assign `hub.Context = testContext`, set `context.Connection.RemoteIpAddress = IPAddress.Parse("10.0.0.1")` and a Basic header for `alice:wrong`. Tests:
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task OnConnectedAsync_FailedBinds_FeedTheSharedLoginThrottle()
|
||||
{
|
||||
// 5 failed hub connects for alice@10.0.0.1 must arm the same lockout the
|
||||
// /management and /auth surfaces consult (arch-review R2 N1).
|
||||
for (var i = 0; i < 5; i++) await ConnectAsync("alice", "wrong"); // each Aborted
|
||||
Assert.True(_throttle.IsLockedOut("alice", "10.0.0.1"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task OnConnectedAsync_LockedOutKey_AbortsWithoutContactingLdap()
|
||||
{
|
||||
for (var i = 0; i < 5; i++) _throttle.RecordFailure("alice", "10.0.0.1");
|
||||
_ldap.ClearReceivedCalls();
|
||||
var ctx = await ConnectAsync("alice", "whatever");
|
||||
Assert.True(ctx.Aborted);
|
||||
await _ldap.DidNotReceiveWithAnyArgs().AuthenticateAsync(default!, default!, default);
|
||||
}
|
||||
```
|
||||
(`ConnectAsync` = build hub + context, `await hub.OnConnectedAsync()`, return the fake context.)
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests --filter DebugStreamHub` — new tests FAIL (the hub binds directly; the throttle never locks and LDAP is contacted while "locked").
|
||||
3. Implement — in `OnConnectedAsync`, delete lines 78–137 (the `TryDisableLoginUser` block, the Basic-decode block, the direct `ILdapAuthService` bind, and the `RoleMapper` call) and replace with:
|
||||
```csharp
|
||||
// Authenticate via the SAME shared Basic→LDAP flow as POST /management and the
|
||||
// audit REST (arch-review R2 N1): ManagementAuthenticator applies the DisableLogin
|
||||
// bypass, the LoginThrottle lockout check BEFORE the bind, and RecordFailure/
|
||||
// RecordSuccess bookkeeping — so hub connection attempts can no longer be used as
|
||||
// an unthrottled LDAP-bind oracle, and hub failures feed the shared lockout.
|
||||
var outcome = await ManagementAuthenticator.AuthenticateAsync(httpContext);
|
||||
if (outcome.User is null)
|
||||
{
|
||||
_logger.LogWarning("DebugStreamHub connection rejected: authentication failed or throttled");
|
||||
Context.Abort();
|
||||
return;
|
||||
}
|
||||
var user = outcome.User;
|
||||
|
||||
// Role check — Deployer role required (unchanged policy).
|
||||
if (!user.Roles.Contains(Roles.Deployer))
|
||||
{
|
||||
_logger.LogWarning("DebugStreamHub connection rejected: {Username} lacks Deployer role", user.Username);
|
||||
Context.Abort();
|
||||
return;
|
||||
}
|
||||
|
||||
// Persist the resolved identity on the connection so per-instance site-scope
|
||||
// enforcement can be applied to SubscribeInstance calls. PermittedSiteIds is the
|
||||
// authenticator's normalized form (empty == system-wide), same as every other surface.
|
||||
Context.Items[RolesKey] = user.Roles;
|
||||
Context.Items[PermittedSiteIdsKey] = user.PermittedSiteIds;
|
||||
|
||||
_logger.LogInformation("DebugStreamHub connection established for {Username}", user.Username);
|
||||
await base.OnConnectedAsync();
|
||||
```
|
||||
Drop the now-unused `using System.Text;` / `ZB.MOM.WW.Auth.Abstractions.Ldap` imports if nothing else uses them.
|
||||
4. Run the filter — PASS. Run the full project: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests` (the five `IsInstanceAccessAllowed` tests and `ManagementAuthenticatorTests` must stay green).
|
||||
5. Fix `Component-Security.md:208`: the claim was false while the hub bound directly — with this change it becomes true. Reword the bullet so it *describes the mechanism* rather than asserting completeness by fiat, e.g.: "…and the HTTP-Basic management/CLI surfaces fronted by `ManagementAuthenticator` — `POST /management`, the audit REST endpoints, **and the debug-stream hub (`DebugStreamHub.OnConnectedAsync` delegates its whole credential path to `ManagementAuthenticator.AuthenticateAsync`, closing the round-2 N1 gap where the hub bound LDAP directly and bypassed the throttle)**. No bind happens outside these paths."
|
||||
6. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ManagementService/DebugStreamHub.cs tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/DebugStreamHubTests.cs docs/requirements/Component-Security.md && git commit -m "fix(security): throttle DebugStreamHub LDAP bind via ManagementAuthenticator + correct coverage doc (plan R2-07 T1)"`
|
||||
|
||||
### Task 2: `ForwardedHeadersSetup` — trusted-proxy client-IP resolution for the throttle keys
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1, 4, 5, 7, 10, 11, 12
|
||||
**Files:**
|
||||
- Create: `src/ZB.MOM.WW.ScadaBridge.Security/ForwardedHeadersSetup.cs` (options class + pure builder)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Program.cs` (central pipeline — insert immediately before `app.UseWebSockets()` at line 340)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.Security.Tests/ForwardedHeadersSetupTests.cs` (create)
|
||||
|
||||
Every throttle surface keys on `context.Connection.RemoteIpAddress` (`ManagementAuthenticator.cs:111`, `AuthEndpoints.cs:39, :126`), and the repo has zero `UseForwardedHeaders` hits — behind Traefik every client shares the proxy IP and the key degenerates to `username|<proxy-ip>` (N2). Fix: honor `X-Forwarded-For` **from the trusted proxy only**.
|
||||
|
||||
1. Failing tests:
|
||||
```csharp
|
||||
using Microsoft.AspNetCore.HttpOverrides;
|
||||
using ZB.MOM.WW.ScadaBridge.Security;
|
||||
using Xunit;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Security.Tests;
|
||||
|
||||
public class ForwardedHeadersSetupTests
|
||||
{
|
||||
[Fact]
|
||||
public void Build_Disabled_ReturnsNull()
|
||||
=> Assert.Null(ForwardedHeadersSetup.Build(new ForwardedHeadersSecurityOptions { Enabled = false }));
|
||||
|
||||
[Fact]
|
||||
public void Build_Enabled_ParsesProxiesAndNetworks_AndClearsDefaults()
|
||||
{
|
||||
var built = ForwardedHeadersSetup.Build(new ForwardedHeadersSecurityOptions
|
||||
{
|
||||
Enabled = true,
|
||||
KnownProxies = ["10.5.0.9"],
|
||||
KnownNetworks = ["172.16.0.0/12"],
|
||||
})!;
|
||||
Assert.Equal(ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto, built.ForwardedHeaders);
|
||||
Assert.Equal(1, built.ForwardLimit);
|
||||
Assert.Single(built.KnownProxies); // ONLY the configured proxy —
|
||||
Assert.DoesNotContain(System.Net.IPAddress.IPv6Loopback, // the loopback default is cleared
|
||||
built.KnownProxies);
|
||||
Assert.Single(built.KnownNetworks);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData("not-an-ip", "172.16.0.0/12")]
|
||||
[InlineData("10.5.0.9", "not-a-cidr")]
|
||||
public void Build_MalformedEntry_ThrowsAtStartup(string proxy, string network)
|
||||
=> Assert.ThrowsAny<Exception>(() => ForwardedHeadersSetup.Build(
|
||||
new ForwardedHeadersSecurityOptions { Enabled = true, KnownProxies = [proxy], KnownNetworks = [network] }));
|
||||
}
|
||||
```
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Security.Tests --filter ForwardedHeadersSetup` — FAIL (types missing).
|
||||
3. Implement `ForwardedHeadersSetup.cs`:
|
||||
```csharp
|
||||
using Microsoft.AspNetCore.Builder;
|
||||
using Microsoft.AspNetCore.HttpOverrides;
|
||||
using System.Net;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Security;
|
||||
|
||||
/// <summary>
|
||||
/// Config shape for trusted-proxy forwarded-header handling
|
||||
/// (<c>ScadaBridge:Security:ForwardedHeaders</c>). Disabled by default: a deployment
|
||||
/// with no reverse proxy must NOT honor X-Forwarded-For (a client could spoof its
|
||||
/// throttle key). The documented Traefik topologies enable it with the docker
|
||||
/// network range so LoginThrottle keys on the REAL client IP (arch-review R2 N2).
|
||||
/// </summary>
|
||||
public sealed class ForwardedHeadersSecurityOptions
|
||||
{
|
||||
public const string SectionName = "ScadaBridge:Security:ForwardedHeaders";
|
||||
public bool Enabled { get; set; }
|
||||
/// <summary>Exact proxy IPs allowed to supply X-Forwarded-For.</summary>
|
||||
public string[] KnownProxies { get; set; } = [];
|
||||
/// <summary>CIDR networks allowed to supply X-Forwarded-For (e.g. "172.16.0.0/12").</summary>
|
||||
public string[] KnownNetworks { get; set; } = [];
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Builds the <see cref="ForwardedHeadersOptions"/> for <c>app.UseForwardedHeaders</c>.
|
||||
/// Pure and fail-fast: malformed entries throw at startup rather than silently
|
||||
/// trusting nothing/everything. Defaults (loopback) are CLEARED — only the
|
||||
/// explicitly configured proxies/networks are trusted, and ForwardLimit=1 means
|
||||
/// only the entry the trusted proxy itself appended is honored.
|
||||
/// </summary>
|
||||
public static class ForwardedHeadersSetup
|
||||
{
|
||||
public static ForwardedHeadersOptions? Build(ForwardedHeadersSecurityOptions options)
|
||||
{
|
||||
if (!options.Enabled) return null;
|
||||
var built = new ForwardedHeadersOptions
|
||||
{
|
||||
ForwardedHeaders = ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto,
|
||||
ForwardLimit = 1,
|
||||
};
|
||||
built.KnownProxies.Clear();
|
||||
built.KnownNetworks.Clear();
|
||||
foreach (var proxy in options.KnownProxies)
|
||||
built.KnownProxies.Add(IPAddress.Parse(proxy));
|
||||
foreach (var network in options.KnownNetworks)
|
||||
{
|
||||
var parts = network.Split('/');
|
||||
built.KnownNetworks.Add(new(IPAddress.Parse(parts[0]), int.Parse(parts[1])));
|
||||
}
|
||||
return built;
|
||||
}
|
||||
}
|
||||
```
|
||||
(`KnownNetworks` element type: use whatever the installed ASP.NET Core surface expects — `Microsoft.AspNetCore.HttpOverrides.IPNetwork` historically, `System.Net.IPNetwork` on newer TFMs; the collection initializer `new(...)` resolves either. Match the compiler.)
|
||||
4. Wire in `Host/Program.cs` — insert immediately **before** `app.UseWebSockets();` (line 340), first middleware in the central pipeline so everything downstream (auth endpoints, `ManagementAuthenticator`, hub) reads the substituted `Connection.RemoteIpAddress`:
|
||||
```csharp
|
||||
// Trusted-proxy forwarded headers (arch-review R2 N2): behind Traefik every client
|
||||
// shares the proxy's IP, collapsing the LoginThrottle's {username}|{ip} keys onto
|
||||
// one address (per-IP isolation gone + cheap username-lockout DoS). When enabled,
|
||||
// X-Forwarded-For from the CONFIGURED proxies only is honored. MUST run first —
|
||||
// everything downstream keys on Connection.RemoteIpAddress.
|
||||
var forwardedOptions = builder.Configuration
|
||||
.GetSection(ZB.MOM.WW.ScadaBridge.Security.ForwardedHeadersSecurityOptions.SectionName)
|
||||
.Get<ZB.MOM.WW.ScadaBridge.Security.ForwardedHeadersSecurityOptions>() ?? new();
|
||||
if (ZB.MOM.WW.ScadaBridge.Security.ForwardedHeadersSetup.Build(forwardedOptions) is { } fho)
|
||||
app.UseForwardedHeaders(fho);
|
||||
```
|
||||
5. Run the filter — PASS; `dotnet build ZB.MOM.WW.ScadaBridge.slnx` — clean; `dotnet test tests/ZB.MOM.WW.ScadaBridge.Security.Tests` — no regressions.
|
||||
6. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Security/ForwardedHeadersSetup.cs src/ZB.MOM.WW.ScadaBridge.Host/Program.cs tests/ZB.MOM.WW.ScadaBridge.Security.Tests/ForwardedHeadersSetupTests.cs && git commit -m "fix(security): trusted-proxy ForwardedHeaders so LoginThrottle keys on the real client IP (plan R2-07 T2)"`
|
||||
|
||||
### Task 3: Ship the ForwardedHeaders config to the Traefik topologies + document the lockout-DoS trade-off
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** 1, 4, 5, 7, 10, 11, 12 (waits on Task 2)
|
||||
**Files:**
|
||||
- Modify: `docker/central-node-a/appsettings.Central.json`, `docker/central-node-b/appsettings.Central.json` (inside the existing `"Security"` object)
|
||||
- Modify: `docker-env2/central-node-a/appsettings.Central.json`, `docker-env2/central-node-b/appsettings.Central.json` (same)
|
||||
- Modify: `deploy/wonder-app-vd03/appsettings.Central.json` (explicit disabled block + comment; no proxy fronts that host)
|
||||
- Modify: `docs/requirements/Component-Security.md` (§Login throttling — key derivation + new trade-off paragraph)
|
||||
- Modify: `tests/ZB.MOM.WW.ScadaBridge.Security.Tests/LoginThrottleTests.cs` (comment only, line 37)
|
||||
|
||||
1. Add to both docker central nodes' `"Security"` section (site nodes don't serve the UI/management surface — untouched):
|
||||
```json
|
||||
"ForwardedHeaders": {
|
||||
"_comment": "Traefik fronts this node on the external scadabridge-net docker network. Trust X-Forwarded-For from the docker bridge address pool so LoginThrottle keys on the real client IP (arch-review R2 N2). Narrow to the Traefik container IP if the network is pinned.",
|
||||
"Enabled": true,
|
||||
"KnownNetworks": [ "172.16.0.0/12" ]
|
||||
}
|
||||
```
|
||||
Same block in both `docker-env2` central nodes. In `deploy/wonder-app-vd03/appsettings.Central.json` add the block with `"Enabled": false` and a comment: "No reverse proxy fronts this host today. If one is introduced, set Enabled=true and list ONLY the proxy's IP in KnownProxies — never enable without it, or clients can spoof their throttle key."
|
||||
2. Validate all five JSONs: `for f in docker/central-node-{a,b}/appsettings.Central.json docker-env2/central-node-{a,b}/appsettings.Central.json deploy/wonder-app-vd03/appsettings.Central.json; do python3 -c "import json,sys;json.load(open(sys.argv[1]))" "$f" || echo "BAD $f"; done`
|
||||
3. `Component-Security.md` §Login throttling — update the "Key and window" bullet: the IP half of the key is the **forwarded-header-resolved client IP** when `ScadaBridge:Security:ForwardedHeaders` names a trusted proxy, otherwise the raw connection peer. Then append a new bullet documenting the trade-off (this text is the deliverable — keep the substance):
|
||||
- **Proxy topologies and the username-lockout DoS.** The key is deliberately `{username}|{IP}`, not `{username}` alone: keying on username alone would let any network-adjacent actor lock any operator out of a SCADA control surface with five wrong passwords, repeatable indefinitely. That isolation only exists when the throttle sees real client IPs — behind a proxy **without** ForwardedHeaders trust, all clients collapse onto the proxy's IP and the DoS returns. Mitigations: (a) the shipped Traefik topologies enable trusted-proxy ForwardedHeaders (so per-IP isolation is real in the documented deployment); (b) the lockout window is short by default (`LoginLockoutMinutes` = 5) and never touches the directory account itself; (c) residual risk — an attacker spraying from their *own* IP locking out a victim username *at that IP only* — is the throttle working as designed. A success-path bypass ("a correct password unlocks") was considered and rejected: it would let an attacker who has the password bypass the lockout entirely, and it converts the throttle into a password oracle during the lockout window.
|
||||
4. `LoginThrottleTests.cs:37` — extend the `// per username+IP key` comment: `// per username+IP key — real isolation in production requires the trusted-proxy ForwardedHeaders config (R2 N2); without it all clients share the proxy IP`.
|
||||
5. `dotnet test tests/ZB.MOM.WW.ScadaBridge.Security.Tests --filter LoginThrottle` — still green (comment-only). Optional smoke after the next `bash docker/deploy.sh`: `docker logs scadabridge-central-a` shows no ForwardedHeaders startup fault, and a failed CLI login from the host still 401s (not 429) on the first attempt.
|
||||
6. Commit: `git add docker docker-env2 deploy/wonder-app-vd03/appsettings.Central.json docs/requirements/Component-Security.md tests/ZB.MOM.WW.ScadaBridge.Security.Tests/LoginThrottleTests.cs && git commit -m "fix(security): ship trusted-proxy ForwardedHeaders config for Traefik topologies + document lockout-DoS trade-off (plan R2-07 T3)"`
|
||||
|
||||
### Task 4: Enforce site scope on secured-write submit / approve / reject
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1, 2, 3, 5, 7, 10, 11, 12 (ManagementActor.cs — single-writer lane; run before Task 6)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs` (`HandleSubmitSecuredWrite` :1243-1283, `HandleApproveSecuredWrite` :1285-1440, `HandleRejectSecuredWrite` :1442-1479)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/SecuredWriteHandlerTests.cs`
|
||||
|
||||
None of the four secured-write handlers checks `user.PermittedSiteIds` (verified: zero `EnforceSiteScope` occurrences in :1243-1510), while `Component-Security.md:141` says scoping "is layered on at the LDAP-mapping level" — a Verifier scoped to site 3 can today approve a **device write** to any site (N3, the same class C2 closed for `DeployArtifacts`). Ruling: **code moves, not the doc** — enforce.
|
||||
|
||||
1. Add a scoped-envelope helper next to the existing `Envelope` helper (`SecuredWriteHandlerTests.cs:151-153`):
|
||||
```csharp
|
||||
private static ManagementEnvelope ScopedEnvelope(object command, string username, string[] permittedSiteIds, params string[] roles) =>
|
||||
new(new AuthenticatedUser(username, username, roles, permittedSiteIds), command, Guid.NewGuid().ToString("N"));
|
||||
```
|
||||
Write the failing negative (and positive) tests — every security task needs the out-of-scope rejection asserted:
|
||||
```csharp
|
||||
[Fact]
|
||||
public void Submit_SiteScopedOperator_OutOfScopeSite_Unauthorized_NoRowInserted()
|
||||
{
|
||||
SeedSiteWithConnection(1, "SITE1", "Gw1", "MxGateway");
|
||||
var actor = CreateActor();
|
||||
actor.Tell(ScopedEnvelope(
|
||||
new SubmitSecuredWriteCommand("SITE1", "Gw1", "Tag.A", "true", "Boolean", "go"),
|
||||
"alice", ["3"], "Operator"));
|
||||
ExpectMsg<ManagementUnauthorized>(TimeSpan.FromSeconds(5));
|
||||
_securedWriteRepo.DidNotReceiveWithAnyArgs().AddAsync(default!, default);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Submit_SiteScopedOperator_InScopeSite_Succeeds()
|
||||
{
|
||||
SeedSiteWithConnection(1, "SITE1", "Gw1", "MxGateway");
|
||||
var actor = CreateActor();
|
||||
actor.Tell(ScopedEnvelope(
|
||||
new SubmitSecuredWriteCommand("SITE1", "Gw1", "Tag.A", "true", "Boolean", "go"),
|
||||
"alice", ["1"], "Operator"));
|
||||
ExpectMsg<ManagementSuccess>(TimeSpan.FromSeconds(5));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Approve_SiteScopedVerifier_OutOfScopeSite_Unauthorized_NeverRelays()
|
||||
{
|
||||
SeedSiteWithConnection(1, "SITE1", "Gw1", "MxGateway");
|
||||
_securedWriteRepo.GetAsync(42, Arg.Any<CancellationToken>()).Returns(new PendingSecuredWrite
|
||||
{
|
||||
Id = 42, SiteId = "SITE1", ConnectionName = "Gw1", TagPath = "Tag.A",
|
||||
ValueJson = "true", ValueType = "Boolean", Status = "Pending",
|
||||
OperatorUser = "alice", SubmittedAtUtc = DateTime.UtcNow,
|
||||
});
|
||||
var actor = CreateActor();
|
||||
actor.Tell(ScopedEnvelope(new ApproveSecuredWriteCommand(42, "ok"), "bob", ["3"], "Verifier"));
|
||||
ExpectMsg<ManagementUnauthorized>(TimeSpan.FromSeconds(5));
|
||||
_securedWriteRepo.DidNotReceiveWithAnyArgs().TryMarkApprovedAsync(default, default!, default, default, default);
|
||||
Assert.Equal(0, _comms.CallCount); // the device write is NEVER relayed
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Reject_SiteScopedVerifier_OutOfScopeSite_Unauthorized() { /* same seed, RejectSecuredWriteCommand(42, "no") → ManagementUnauthorized; UpdateAsync not received */ }
|
||||
|
||||
[Fact]
|
||||
public void Approve_ScopedAdministrator_BypassesSiteScope() { /* ScopedEnvelope(..., "carol", ["3"], "Verifier", "Administrator") on the SITE1 row → NOT ManagementUnauthorized (EnforceSiteScope admin bypass, ManagementActor.cs:499) */ }
|
||||
```
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests --filter SecuredWrite` — new tests FAIL (no scope check exists).
|
||||
3. Implement — three insertions, all **before** any state mutation or audit emission:
|
||||
- `HandleSubmitSecuredWrite`: the site entity is already loaded at :1247-1248, so use the cheap overload right after it:
|
||||
```csharp
|
||||
// Site scope (arch-review R2 N3): an LDAP-mapping-scoped Operator may only
|
||||
// submit device writes to their permitted sites — same rule C2 applies to
|
||||
// DeployArtifacts, on a higher-consequence path. Checked before the row exists.
|
||||
EnforceSiteScope(user, site.Id);
|
||||
```
|
||||
- `HandleApproveSecuredWrite`: after the row load + Pending check (:1289-1294), **before** the TTL/self-approval/CAS chain:
|
||||
```csharp
|
||||
// Site scope (arch-review R2 N3): the approving Verifier must be permitted on
|
||||
// the row's target site — checked BEFORE the TTL/self-approval/CAS chain so an
|
||||
// out-of-scope verifier can neither consume the Pending transition nor relay.
|
||||
await EnforceSiteScopeForIdentifier(sp, user, row.SiteId);
|
||||
```
|
||||
- `HandleRejectSecuredWrite`: same line after :1447-1451, against `entity.SiteId`.
|
||||
4. Run the filter — PASS. Run the full project (`dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests`) — the existing submit/approve/reject tests use the system-wide `Envelope` helper (`PermittedSiteIds` empty), so `EnforceSiteScope` no-ops and they stay green. `RequiredRoleMatrixTests` is untouched: no role-set changes, only in-handler scope enforcement (the frozen table freezes *roles*, not scope — see its header comment "Do not regenerate it mechanically"). If it fails, STOP: something else changed.
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/SecuredWriteHandlerTests.cs && git commit -m "fix(security): enforce LDAP-mapping site scope on secured-write submit/approve/reject (plan R2-07 T4)"`
|
||||
|
||||
### Task 5: `ISecuredWriteRepository` — additive permitted-sites filter for scoped listing
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** 1, 2, 3, 4, 7, 10, 11, 12
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/ISecuredWriteRepository.cs` (`QueryAsync` :51, `CountAsync` :112)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SecuredWriteRepository.cs` (`QueryAsync` :55, `CountAsync` :131)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/SecuredWriteRepositoryTests.cs` (extend)
|
||||
|
||||
Task 6 needs to scope the *unfiltered* list for a site-scoped caller at the query level (so rows **and** `totalCount` are both scoped, and paging stays correct). Give both query methods an additive optional filter.
|
||||
|
||||
1. Failing tests (mirror the existing `SecuredWriteRepositoryTests` seeding — rows across site identifiers `"SITE1"`, `"SITE2"`, `"SITE3"`):
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task QueryAsync_PermittedSiteIds_ReturnsOnlyPermittedRows()
|
||||
{
|
||||
// seed 2 rows on SITE1, 1 on SITE2, 1 on SITE3 (existing seeding helper)
|
||||
var rows = await _repo.QueryAsync(status: null, siteId: null, skip: 0, take: 100,
|
||||
permittedSiteIds: new[] { "SITE1", "SITE3" });
|
||||
Assert.Equal(3, rows.Count);
|
||||
Assert.DoesNotContain(rows, r => r.SiteId == "SITE2");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task CountAsync_PermittedSiteIds_CountsOnlyPermittedRows()
|
||||
{
|
||||
var count = await _repo.CountAsync(status: null, siteId: null,
|
||||
permittedSiteIds: new[] { "SITE1", "SITE3" });
|
||||
Assert.Equal(3, count);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task QueryAsync_NullPermittedSiteIds_IsUnfiltered_BackCompat()
|
||||
{ /* null → all 4 rows, exactly today's behavior */ }
|
||||
```
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter SecuredWriteRepository` — FAIL (no such parameter).
|
||||
3. Implement — additive optional parameter before the `CancellationToken` on both members (interface + impl):
|
||||
```csharp
|
||||
Task<IReadOnlyList<PendingSecuredWrite>> QueryAsync(
|
||||
string? status, string? siteId, int skip, int take,
|
||||
IReadOnlyCollection<string>? permittedSiteIds = null, CancellationToken ct = default);
|
||||
|
||||
Task<int> CountAsync(
|
||||
string? status, string? siteId,
|
||||
IReadOnlyCollection<string>? permittedSiteIds = null, CancellationToken ct = default);
|
||||
```
|
||||
xmldoc: "`permittedSiteIds` — when non-null, only rows whose `SiteId` (site identifier) is in the set match; `null` matches every site. Applied IN ADDITION to `siteId`. Used by the ManagementActor to constrain an unfiltered list to a site-scoped caller's permitted sites at the query level (arch-review R2 N3), so paging and totals stay correct." EF impl, in both methods' filter composition:
|
||||
```csharp
|
||||
if (permittedSiteIds is not null)
|
||||
query = query.Where(w => permittedSiteIds.Contains(w.SiteId));
|
||||
```
|
||||
4. Run the filter — PASS; full project run (existing positional callers compile unchanged — the parameter is trailing-optional).
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/ISecuredWriteRepository.cs src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SecuredWriteRepository.cs tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/SecuredWriteRepositoryTests.cs && git commit -m "fix(security): additive permitted-sites filter on secured-write query/count (plan R2-07 T5)"`
|
||||
|
||||
### Task 6: Scope-filter `HandleListSecuredWrites` + amend `Component-Security.md:141` + matrix verification
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** none (ManagementActor.cs — single-writer lane; waits on Tasks 4 and 5)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs` (`HandleListSecuredWrites` :1481-1509; dispatch arm :470)
|
||||
- Modify: `docs/requirements/Component-Security.md` (line 141 — the "any site scoping is layered on at the LDAP-mapping level" sentence)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/SecuredWriteHandlerTests.cs`
|
||||
|
||||
1. Failing tests:
|
||||
```csharp
|
||||
[Fact]
|
||||
public void List_SiteScopedReader_ExplicitOutOfScopeSiteFilter_Unauthorized()
|
||||
{
|
||||
SeedSiteWithConnection(1, "SITE1", "Gw1", "MxGateway");
|
||||
var actor = CreateActor();
|
||||
actor.Tell(ScopedEnvelope(new ListSecuredWritesCommand(null, "SITE1"), "bob", ["3"], "Verifier"));
|
||||
ExpectMsg<ManagementUnauthorized>(TimeSpan.FromSeconds(5));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void List_SiteScopedReader_Unfiltered_QueriesOnlyPermittedIdentifiers()
|
||||
{
|
||||
_siteRepo.GetAllSitesAsync(Arg.Any<CancellationToken>()).Returns(new List<Site>
|
||||
{
|
||||
new("Site1", "SITE1") { Id = 1 },
|
||||
new("Site3", "SITE3") { Id = 3 },
|
||||
});
|
||||
_securedWriteRepo.QueryAsync(null, null, 0, Arg.Any<int>(),
|
||||
Arg.Any<IReadOnlyCollection<string>?>(), Arg.Any<CancellationToken>())
|
||||
.Returns(new List<PendingSecuredWrite>());
|
||||
var actor = CreateActor();
|
||||
actor.Tell(ScopedEnvelope(new ListSecuredWritesCommand(null, null), "bob", ["3"], "Verifier"));
|
||||
ExpectMsg<ManagementSuccess>(TimeSpan.FromSeconds(5));
|
||||
_securedWriteRepo.Received(1).QueryAsync(null, null, 0, Arg.Any<int>(),
|
||||
Arg.Is<IReadOnlyCollection<string>?>(s => s != null && s.Single() == "SITE3"),
|
||||
Arg.Any<CancellationToken>());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void List_SystemWideReader_PassesNullPermittedFilter() { /* Envelope(...) → QueryAsync received with permittedSiteIds == null */ }
|
||||
```
|
||||
(Match the real `ListSecuredWritesCommand` parameter order — read `SecuredWriteCommands.cs:63`: `(Status, SiteId, Skip, Take)`.)
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests --filter List_SiteScoped` — FAIL.
|
||||
3. Implement — change the handler signature to take the user and the dispatch arm at :470 from `HandleListSecuredWrites(sp, cmd)` to `HandleListSecuredWrites(sp, cmd, user)`:
|
||||
```csharp
|
||||
private static async Task<object?> HandleListSecuredWrites(
|
||||
IServiceProvider sp, ListSecuredWritesCommand cmd, AuthenticatedUser user)
|
||||
{
|
||||
// Site scoping (arch-review R2 N3): an explicit SiteId filter is scope-checked
|
||||
// like every other identifier-keyed command; an UNFILTERED list from a
|
||||
// site-scoped non-admin caller is constrained to their permitted sites at the
|
||||
// query level (rows AND totalCount both scoped — see ISecuredWriteRepository).
|
||||
IReadOnlyCollection<string>? permittedIdentifiers = null;
|
||||
if (cmd.SiteId is not null)
|
||||
{
|
||||
await EnforceSiteScopeForIdentifier(sp, user, cmd.SiteId);
|
||||
}
|
||||
else if (user.PermittedSiteIds.Length > 0
|
||||
&& !user.Roles.Contains(Roles.Administrator, StringComparer.OrdinalIgnoreCase))
|
||||
{
|
||||
var siteRepo = sp.GetRequiredService<ISiteRepository>();
|
||||
var permitted = new HashSet<string>(user.PermittedSiteIds);
|
||||
permittedIdentifiers = (await siteRepo.GetAllSitesAsync())
|
||||
.Where(s => permitted.Contains(s.Id.ToString()))
|
||||
.Select(s => s.SiteIdentifier)
|
||||
.ToList();
|
||||
}
|
||||
|
||||
var repo = sp.GetRequiredService<ISecuredWriteRepository>();
|
||||
var take = Math.Clamp(cmd.Take, 1, 500);
|
||||
var skip = Math.Max(0, cmd.Skip);
|
||||
var rows = await repo.QueryAsync(cmd.Status, cmd.SiteId, skip, take, permittedIdentifiers);
|
||||
var totalCount = await repo.CountAsync(cmd.Status, cmd.SiteId, permittedIdentifiers);
|
||||
// ... (opportunistic expiry sweep unchanged)
|
||||
```
|
||||
4. Run the filter — PASS. Then the lock-in suites — the frozen table needs **no regeneration** (role sets untouched; per its header, the table freezes the `GetRequiredRoles` switch, which this task does not edit) and the dispatch arm still reaches a handler:
|
||||
`dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests --filter "RequiredRoleMatrixTests|DispatchCoverageTests"` — green. If `RequiredRoleMatrixTests` DOES fail, follow its documented update procedure (hand-author the changed entry against the switch + cross-check `Component-ManagementService.md` §Authorization — never regenerate mechanically). Full project run.
|
||||
5. Amend `Component-Security.md:141` — replace "Both are coarse global roles like the others; any site scoping is layered on at the LDAP-mapping level." with: "Both are coarse global roles like the others. **Site scoping from the LDAP mapping is enforced server-side on every secured-write command** (arch-review R2 N3): submit checks the target site, approve/reject check the row's site (before the TTL/self-approval/CAS chain), and the list constrains a scoped caller to their permitted sites — mirroring the deployment commands' `EnforceSiteScope` rule. Administrators and system-wide principals (empty permitted-site set) are unrestricted."
|
||||
6. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/SecuredWriteHandlerTests.cs docs/requirements/Component-Security.md && git commit -m "fix(security): scope-filter secured-write listing + align spec with enforced site scoping (plan R2-07 T6)"`
|
||||
|
||||
### Task 7: `AlarmSummary.RefreshAsync` stale-site guard — never display cross-site data
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** 1, 2, 3, 4, 5, 10, 11, 12 (AlarmSummary.razor lane — run before Tasks 8, 9)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/AlarmSummary.razor` (`RefreshAsync` :283-308)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Monitoring/AlarmSummaryRenderTests.cs`
|
||||
|
||||
`RefreshAsync` captures `siteId` at entry (:285), awaits the multi-second fan-out (:293), then unconditionally assigns `_rows`/`_notReporting`/`_rollup` (:294-296) — a site-A poll completing after a switch to site B overwrites B's alarms with A's, on an operator alarm page (N4). The live path already has exactly this guard (`OnLiveAlarmsChanged`, :374).
|
||||
|
||||
1. Failing bUnit test (the harness's constructor stubs `GetSiteAlarmsAsync(Arg.Any<int>, ...)`; override per-site inside the test):
|
||||
```csharp
|
||||
[Fact]
|
||||
public void InFlightPollForPreviousSite_DoesNotOverwriteNewSitesRows()
|
||||
{
|
||||
// Site 1's fan-out hangs on a TCS; site 2 answers immediately (arch-review R2 N4).
|
||||
var site1Tcs = new TaskCompletionSource<AlarmSummaryResult>();
|
||||
_summary.GetSiteAlarmsAsync(1, Arg.Any<CancellationToken>()).Returns(site1Tcs.Task);
|
||||
_summary.GetSiteAlarmsAsync(2, Arg.Any<CancellationToken>()).Returns(Task.FromResult(
|
||||
new AlarmSummaryResult(
|
||||
new List<AlarmSummaryRow>
|
||||
{
|
||||
new("Site2Instance", new AlarmStateChanged("Site2Instance", "S2-alarm", AlarmState.Active, 100, DateTimeOffset.UtcNow)),
|
||||
},
|
||||
Array.Empty<string>())));
|
||||
|
||||
var cut = Render<AlarmSummaryPage>();
|
||||
cut.Find("[data-test='alarm-summary-site']").Change("1"); // poll in flight, hung
|
||||
cut.Find("[data-test='alarm-summary-site']").Change("2"); // site 2 renders
|
||||
cut.WaitForAssertion(() => Assert.Equal("Site2Instance", FirstRowInstance(cut)));
|
||||
|
||||
// The stale site-1 fan-out now completes — it must be dropped, not rendered.
|
||||
site1Tcs.SetResult(new AlarmSummaryResult(
|
||||
new List<AlarmSummaryRow>
|
||||
{
|
||||
new("StaleSite1", new AlarmStateChanged("StaleSite1", "S1-alarm", AlarmState.Active, 999, DateTimeOffset.UtcNow)),
|
||||
},
|
||||
new[] { "StaleNotReporting" }));
|
||||
|
||||
cut.WaitForAssertion(() =>
|
||||
{
|
||||
Assert.Equal("Site2Instance", FirstRowInstance(cut));
|
||||
Assert.DoesNotContain("StaleNotReporting", cut.Markup);
|
||||
});
|
||||
}
|
||||
```
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests --filter AlarmSummary` — new test FAILS (StaleSite1 overwrites the page).
|
||||
3. Implement — one guard in `RefreshAsync`, immediately after the await at :293:
|
||||
```csharp
|
||||
var result = await AlarmSummaryService.GetSiteAlarmsAsync(siteId);
|
||||
// Stale-site guard (arch-review R2 N4): the operator may have switched sites
|
||||
// while this fan-out was in flight — drop the result rather than labeling
|
||||
// site A's alarms under site B's picker. Mirrors OnLiveAlarmsChanged (:374).
|
||||
if (_selectedSiteId != siteId)
|
||||
{
|
||||
return;
|
||||
}
|
||||
```
|
||||
(`_loading` is reset in the existing `finally` — correct either way, since a switch re-enters `RefreshAsync` for the new site.)
|
||||
4. Run the filter — PASS, plus the pre-existing AlarmSummary tests stay green.
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/AlarmSummary.razor tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Monitoring/AlarmSummaryRenderTests.cs && git commit -m "fix(ui): drop stale-site poll results on Alarm Summary site switch (plan R2-07 T7)"`
|
||||
|
||||
### Task 8: While live, the poll updates only `_notReporting` — never regresses live rows
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** none (AlarmSummary.razor lane; waits on Task 7)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/AlarmSummary.razor` (`RefreshAsync` :283-308; reconciliation comment block :337-351)
|
||||
- Modify: `docs/requirements/Component-CentralUI.md` (line 194, the Refresh bullet)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Monitoring/AlarmSummaryRenderTests.cs`
|
||||
|
||||
The 15 s poll always rebuilds `_rows` from the fan-out even when the cache is live; a poll whose fan-out started before a live delta lands after it and momentarily reverts the delta (N5). The comment at :344-347 ("the poll simply re-affirms the same snapshot") is wrong — fix both.
|
||||
|
||||
1. Failing bUnit test (uses the manual Refresh button, `data-test="alarm-summary-refresh"`, to drive the poll path deterministically — it invokes the same `RefreshAsync` the timer does):
|
||||
```csharp
|
||||
[Fact]
|
||||
public void PollWhileLive_UpdatesNotReportingOnly_DoesNotRegressLiveRows()
|
||||
{
|
||||
// Poll snapshot = Zeta/Alpha (constructor stub) + a not-reporting instance.
|
||||
_summary.GetSiteAlarmsAsync(Arg.Any<int>(), Arg.Any<CancellationToken>())
|
||||
.Returns(ci => Task.FromResult(new AlarmSummaryResult(
|
||||
new List<AlarmSummaryRow>
|
||||
{
|
||||
new("Zeta", new AlarmStateChanged("Zeta", "Z-alarm", AlarmState.Active, 900, DateTimeOffset.UtcNow)),
|
||||
},
|
||||
new[] { "OfflineInstance" })));
|
||||
|
||||
var cut = RenderWithSiteSelected();
|
||||
|
||||
// Go live with a fresher delta: Gamma only (arch-review R2 N5).
|
||||
_liveCache.PushAlarms(new List<AlarmStateChanged>
|
||||
{
|
||||
new("Gamma", "G-alarm", AlarmState.Active, 300, DateTimeOffset.UtcNow),
|
||||
});
|
||||
cut.WaitForAssertion(() => Assert.Equal("Gamma", FirstRowInstance(cut)));
|
||||
|
||||
// A poll now completes: it owns _notReporting but must NOT rebuild the rows.
|
||||
cut.Find("[data-test='alarm-summary-refresh']").Click();
|
||||
|
||||
cut.WaitForAssertion(() =>
|
||||
{
|
||||
Assert.Contains("OfflineInstance", cut.Markup); // notReporting refreshed
|
||||
Assert.Equal("Gamma", FirstRowInstance(cut)); // live rows NOT regressed
|
||||
Assert.Single(cut.FindAll("tr[data-test='alarm-summary-row']"));
|
||||
});
|
||||
}
|
||||
```
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests --filter AlarmSummary` — FAILS (the poll rebuilds Zeta over Gamma).
|
||||
3. Implement — in `RefreshAsync`, after Task 7's stale-site guard, replace the unconditional assignments:
|
||||
```csharp
|
||||
// _notReporting is the poll's unique authority — the alarm-only live cache
|
||||
// cannot compute it — so it is always refreshed.
|
||||
_notReporting = result.NotReportingInstances;
|
||||
|
||||
// While the cache is live, the live deltas own the row set: a poll whose fan-out
|
||||
// started BEFORE a delta must not land after it and momentarily revert the alarm
|
||||
// state (arch-review R2 N5). When not live (pre-seed / degraded stream / dead
|
||||
// aggregator — see R2 N6), the poll remains the full-rebuild safety net.
|
||||
if (!LiveAlarmCache.IsLive(siteId))
|
||||
{
|
||||
_rows = result.Alarms;
|
||||
_rollup = AlarmSummaryService.ComputeRollup(_rows);
|
||||
}
|
||||
RecomputeVisibleRows();
|
||||
```
|
||||
Rewrite the stale sentence in the reconciliation comment block (:344-347): the poll is "the authority for `_notReporting` and the full-rebuild safety net **only while the cache is not live**; when live it deliberately leaves `_rows` to the delta path".
|
||||
4. Run the filter — PASS. Note interplay checks: `PollFallbackRenders_WhenCacheNotLiveYet` still passes (not live → full rebuild); `OnChangedDelta_RebuildsRenderedRowsFromLiveSnapshot` still passes (live path untouched).
|
||||
5. Update `Component-CentralUI.md:194`: append "…when live, the poll updates only the `NotReporting` list and leaves the row set to the delta path, so a slow fan-out can never momentarily revert a fresher live delta (R2 N5)."
|
||||
6. Commit: `git add src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/AlarmSummary.razor tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Monitoring/AlarmSummaryRenderTests.cs docs/requirements/Component-CentralUI.md && git commit -m "fix(ui): Alarm Summary poll defers row rebuilds to the live path while IsLive (plan R2-07 T8)"`
|
||||
|
||||
### Task 9: House disposal guard on the live callback
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** none (AlarmSummary.razor lane; waits on Task 8)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/AlarmSummary.razor` (`OnLiveAlarmsChanged` :367-382, `Dispose` :507-511)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Monitoring/AlarmSummaryRenderTests.cs` (+ add a `CapturedCallback` accessor to `FakeSiteAlarmLiveCache`)
|
||||
|
||||
`OnLiveAlarmsChanged` does `_ = InvokeAsync(...)` with no `_disposed` flag; a callback racing `Dispose()` can fault the discarded task with `ObjectDisposedException` from `StateHasChanged` on a torn-down renderer (N7). `DebugView.razor` established the pattern (`_disposed` set before teardown + checked before AND inside the marshalled work — :312, :430-433, :606).
|
||||
|
||||
1. Failing test — expose the raw registered callback on the fake so the test can simulate a delta already in flight when `Dispose` runs (bUnit's renderer stays alive after `cut.Instance.Dispose()`, so pre-fix the late callback observably rebuilds the rows):
|
||||
```csharp
|
||||
// In FakeSiteAlarmLiveCache:
|
||||
public Action? CapturedCallback => _onChanged;
|
||||
|
||||
[Fact]
|
||||
public void LiveCallbackRacingDispose_IsDropped_NoRebuildNoThrow()
|
||||
{
|
||||
var cut = RenderWithSiteSelected();
|
||||
_liveCache.PushAlarms(new List<AlarmStateChanged>
|
||||
{
|
||||
new("Gamma", "G-alarm", AlarmState.Active, 300, DateTimeOffset.UtcNow),
|
||||
});
|
||||
cut.WaitForAssertion(() => Assert.Equal("Gamma", FirstRowInstance(cut)));
|
||||
|
||||
// Snapshot the callback BEFORE disposal — this models a delta thread that
|
||||
// already read the delegate when Dispose ran (arch-review R2 N7).
|
||||
var inFlight = _liveCache.CapturedCallback!;
|
||||
cut.Instance.Dispose();
|
||||
|
||||
_liveCache.PushAlarms(new List<AlarmStateChanged>()); // mutate the fake's snapshot
|
||||
var ex = Record.Exception(inFlight);
|
||||
Assert.Null(ex);
|
||||
// The disposed page must not have re-applied the (now empty) live snapshot.
|
||||
Assert.Equal("Gamma", FirstRowInstance(cut));
|
||||
}
|
||||
```
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests --filter AlarmSummary` — FAILS (the late callback re-applies the empty snapshot: zero rows).
|
||||
3. Implement — mirror `DebugView.razor`:
|
||||
```csharp
|
||||
// N7: set BEFORE teardown so a live callback racing Dispose is dropped both
|
||||
// before the InvokeAsync marshal and inside it (mirrors DebugView.razor).
|
||||
private volatile bool _disposed;
|
||||
|
||||
private void OnLiveAlarmsChanged(int siteId)
|
||||
{
|
||||
if (_disposed) return;
|
||||
_ = InvokeAsync(() =>
|
||||
{
|
||||
if (_disposed || _selectedSiteId != siteId || !LiveAlarmCache.IsLive(siteId))
|
||||
{
|
||||
return;
|
||||
}
|
||||
ApplyLiveSnapshot(siteId);
|
||||
StateHasChanged();
|
||||
});
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
_disposed = true;
|
||||
StopTimer();
|
||||
DisposeLiveSubscription();
|
||||
}
|
||||
```
|
||||
4. Run the filter — PASS (including `DisposingComponent_UnsubscribesFromTheLiveCache`).
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/AlarmSummary.razor tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Monitoring/AlarmSummaryRenderTests.cs && git commit -m "fix(ui): disposal guard on Alarm Summary live callback, matching the DebugView pattern (plan R2-07 T9)"`
|
||||
|
||||
### Task 10: Un-stick `IsLive` — deathwatch resets liveness when the aggregator terminates
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1, 2, 3, 4, 5, 7, 11, 12 — **MUTEX: must NOT run concurrently with PLAN-R2-02's `SiteAlarmLiveCacheService` delta-coalescing task** (see "Parallelization & mutex rules"); whichever starts first lands its commit before the other begins
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Communication/SiteAlarmLiveCacheService.cs` (`StartAggregatorAsync` :219-243; new nested watcher actor + `OnAggregatorTerminated`; `SiteEntry` :416-453)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Communication/ISiteAlarmLiveCache.cs` (`IsLive` xmldoc :53-61)
|
||||
- Modify: `docs/requirements/Component-CentralUI.md` (line 194 — "when a stream is unhealthy" wording gains the aggregator-death case)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/SiteAlarmLiveCacheServiceTests.cs`
|
||||
|
||||
`SiteEntry.HasPublished` is set on first publish (:392-393) and never cleared, so `IsLive` (:132-136) reports `true` forever — and `OnSiteChangedAsync` deliberately applies the live snapshot **on top of** the fresh poll when `IsLive` (`AlarmSummary.razor:277-280`), grafting an arbitrarily stale frozen snapshot over correct data if the aggregator died (N6). Fix: deathwatch the aggregator and reset liveness (which also makes the page's poll the authority again, via Task 8's `!IsLive` branch).
|
||||
|
||||
1. Failing tests (the harness runs a real TestKit `ActorSystem` — `SiteAlarmLiveCacheServiceTests` — and the aggregator is a real child at `/user/site-alarm-aggregator-{siteId}-{guid}`):
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task Aggregator_Death_Resets_IsLive_And_Clears_The_Snapshot()
|
||||
{
|
||||
var service = CreateService(TimeSpan.FromMinutes(5), out _);
|
||||
using var sub = service.Subscribe(SiteId, () => { });
|
||||
AwaitAssert(() => Assert.True(service.IsLive(SiteId)), TimeSpan.FromSeconds(10));
|
||||
|
||||
// Kill the aggregator out from under the service (crash-death, NOT a linger stop).
|
||||
var aggregator = await Sys.ActorSelection($"/user/site-alarm-aggregator-{SiteId}-*")
|
||||
.ResolveOne(TimeSpan.FromSeconds(5));
|
||||
Sys.Stop(aggregator);
|
||||
|
||||
AwaitAssert(() =>
|
||||
{
|
||||
Assert.False(service.IsLive(SiteId)); // sticky no more (R2 N6)
|
||||
Assert.Empty(service.GetCurrentAlarms(SiteId)); // frozen snapshot never grafted
|
||||
}, TimeSpan.FromSeconds(10));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Linger_Stop_Does_Not_Resurrect_The_Aggregator()
|
||||
{
|
||||
// Last viewer leaves, linger fires, entry removed: the deathwatch on the
|
||||
// deliberately-stopped actor must be a no-op (entry gone), not a restart.
|
||||
var service = CreateService(TimeSpan.FromMilliseconds(50), out var factory);
|
||||
var sub = service.Subscribe(SiteId, () => { });
|
||||
AwaitAssert(() => Assert.True(service.IsLive(SiteId)), TimeSpan.FromSeconds(10));
|
||||
sub.Dispose();
|
||||
AwaitAssert(() => Assert.False(service.IsLive(SiteId)), TimeSpan.FromSeconds(10));
|
||||
var startsAfterStop = factory.GetOrCreateCount;
|
||||
Thread.Sleep(300);
|
||||
Assert.Equal(startsAfterStop, factory.GetOrCreateCount); // no self-heal restart fired
|
||||
}
|
||||
```
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Communication.Tests --filter SiteAlarmLiveCache` — first test FAILS (`IsLive` stays true after the stop).
|
||||
3. Implement:
|
||||
- In `StartAggregatorAsync`, inside the lock right after `entry.Actor = system.ActorOf(props, ...)` (:238), capture and watch:
|
||||
```csharp
|
||||
var aggregator = entry.Actor;
|
||||
// Deathwatch (arch-review R2 N6): if the aggregator terminates for any reason,
|
||||
// liveness must reset — a dead feed must never keep IsLive == true, or the page
|
||||
// grafts a frozen snapshot over fresh poll data for up to 15 s.
|
||||
system.ActorOf(Props.Create(() => new AggregatorTerminationWatcher(
|
||||
aggregator!, () => OnAggregatorTerminated(entry.SiteId, aggregator!))));
|
||||
```
|
||||
- New members on the service:
|
||||
```csharp
|
||||
/// <summary>Watches one aggregator and fires a callback exactly once on Terminated.</summary>
|
||||
private sealed class AggregatorTerminationWatcher : ReceiveActor
|
||||
{
|
||||
public AggregatorTerminationWatcher(IActorRef watched, Action onTerminated)
|
||||
{
|
||||
Context.Watch(watched);
|
||||
Receive<Terminated>(_ =>
|
||||
{
|
||||
onTerminated();
|
||||
Context.Stop(Self);
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Terminated hook (R2 N6). A DELIBERATE stop (linger fired, entry already removed
|
||||
/// from _sites) is a no-op — the ReferenceEquals guard also ignores a stale watcher
|
||||
/// firing after a newer aggregator replaced the dead one. A crash-death with live
|
||||
/// viewers resets liveness (IsLive → false, snapshot cleared, so the page's poll is
|
||||
/// authoritative again) and arms the existing bounded start-retry so the feature
|
||||
/// self-heals, mirroring StartAggregatorAsync's transient-failure path.
|
||||
/// </summary>
|
||||
private void OnAggregatorTerminated(int siteId, IActorRef deadActor)
|
||||
{
|
||||
lock (_lock)
|
||||
{
|
||||
if (!_sites.TryGetValue(siteId, out var entry) || !ReferenceEquals(entry.Actor, deadActor))
|
||||
return;
|
||||
|
||||
entry.Actor = null;
|
||||
entry.HasPublished = false;
|
||||
entry.Current = Empty;
|
||||
|
||||
if (entry.Subscribers.Count > 0 && !entry.Starting)
|
||||
{
|
||||
entry.Starting = true;
|
||||
entry.StartRetryTimer?.Dispose();
|
||||
entry.StartRetryTimer = new Timer(
|
||||
_ => { _ = StartAggregatorAsync(entry); },
|
||||
state: null,
|
||||
dueTime: _options.LiveAlarmCacheReconcileInterval,
|
||||
period: Timeout.InfiniteTimeSpan);
|
||||
}
|
||||
}
|
||||
_logger.LogWarning(
|
||||
"Live alarm aggregator for site {SiteId} terminated unexpectedly; liveness reset " +
|
||||
"(page falls back to polling) and a restart is armed while viewers remain", siteId);
|
||||
}
|
||||
```
|
||||
- `ISiteAlarmLiveCache.IsLive` xmldoc: replace "true once the site's aggregator has seeded and published at least once" with "true while a **living** aggregator has seeded and published; reverts to false if the aggregator terminates (R2 N6) — a frozen snapshot is never reported as live".
|
||||
4. Run the filter — PASS (all pre-existing lifecycle tests — shared start, linger stop, re-subscribe, cap, self-heal — must stay green; the linger-stop path removes the entry before the actor stops, so the new hook's `TryGetValue` guard keeps it inert there).
|
||||
5. Update `Component-CentralUI.md:194`: "…when a stream is unhealthy, **the aggregator has died (deathwatch resets `IsLive`)**, or a site has not yet seeded, the poll keeps the page fresh…"
|
||||
6. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Communication/SiteAlarmLiveCacheService.cs src/ZB.MOM.WW.ScadaBridge.Communication/ISiteAlarmLiveCache.cs tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/SiteAlarmLiveCacheServiceTests.cs docs/requirements/Component-CentralUI.md && git commit -m "fix(sitestream): deathwatch resets live-cache liveness on aggregator termination (plan R2-07 T10)"`
|
||||
|
||||
### Task 11: Amortize `LoginThrottle.Prune` off the failure hot path
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** 1, 2, 4, 5, 7, 10, 12
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Security/LoginThrottle.cs` (`RecordFailure` :121, `Prune` :146-169; new counter field)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.Security.Tests/LoginThrottleTests.cs`
|
||||
|
||||
`RecordFailure` → `Prune` enumerates all entries (and may `OrderBy` the whole map) on **every** failed bind — during the exact spray the throttle exists for, every request pays an O(n≤10k) scan (N8). Amortize: sweep every Nth failure, plus immediately whenever the map exceeds the hard cap (the cap stays the memory-safety invariant; only the expired-entry housekeeping is batched).
|
||||
|
||||
1. Failing test (needs a diagnostics accessor — `LoginThrottle` already has `InternalsVisibleTo` for Security.Tests via the csproj):
|
||||
```csharp
|
||||
[Fact]
|
||||
public void KeySpray_MapStaysBounded_AtTheHardCap()
|
||||
{
|
||||
var (throttle, _) = Create(); // existing helper: throttle over a fake clock
|
||||
// Spray far past the cap: the over-cap eviction must run on EVERY write once
|
||||
// the cap is exceeded, regardless of the amortized cadence (arch-review R2 N8).
|
||||
for (var i = 0; i < LoginThrottle.MaxTrackedKeys + 500; i++)
|
||||
throttle.RecordFailure($"user{i}", "10.0.0.1");
|
||||
Assert.True(throttle.TrackedKeyCount <= LoginThrottle.MaxTrackedKeys);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ExpiredEntries_AreEventuallySwept_OnTheAmortizedCadence()
|
||||
{
|
||||
var (throttle, clock) = Create();
|
||||
throttle.RecordFailure("stale", "10.0.0.1");
|
||||
clock.Advance(TimeSpan.FromMinutes(10)); // window fully expired
|
||||
for (var i = 0; i < LoginThrottle.PruneEveryNFailures; i++)
|
||||
throttle.RecordFailure($"fresh{i}", "10.0.0.1"); // cadence tick fires within these
|
||||
Assert.False(throttle.IsLockedOut("stale", "10.0.0.1"));
|
||||
Assert.True(throttle.TrackedKeyCount <= LoginThrottle.PruneEveryNFailures); // "stale" swept
|
||||
}
|
||||
```
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Security.Tests --filter LoginThrottle` — FAIL (compile: `TrackedKeyCount`/`PruneEveryNFailures` missing).
|
||||
3. Implement:
|
||||
```csharp
|
||||
/// <summary>Amortization cadence: a full prune sweep runs every Nth failure write
|
||||
/// (plus immediately whenever the map exceeds <see cref="MaxTrackedKeys"/>), so a
|
||||
/// spray does not pay an O(n) scan — and a possible full sort — on every failed
|
||||
/// request (arch-review R2 N8). Lockout SEMANTICS are unaffected: IsLockedOut reads
|
||||
/// window/lockout expiry directly and never depends on pruning.</summary>
|
||||
internal const int PruneEveryNFailures = 64;
|
||||
|
||||
/// <summary>Diagnostics/test accessor: number of currently tracked keys.</summary>
|
||||
internal int TrackedKeyCount => _entries.Count;
|
||||
|
||||
private int _failureWrites;
|
||||
```
|
||||
Replace the unconditional `Prune(now);` at :121 with:
|
||||
```csharp
|
||||
// Amortized prune (R2 N8): every Nth failure, or immediately when over the hard
|
||||
// cap — the cap is the memory-safety invariant and must never wait for the cadence.
|
||||
if (Interlocked.Increment(ref _failureWrites) % PruneEveryNFailures == 0
|
||||
|| _entries.Count > MaxTrackedKeys)
|
||||
{
|
||||
Prune(now);
|
||||
}
|
||||
```
|
||||
4. Run the filter — PASS, and ALL seven pre-existing throttle tests must stay green unchanged (they assert lockout semantics, which never depended on pruning).
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Security/LoginThrottle.cs tests/ZB.MOM.WW.ScadaBridge.Security.Tests/LoginThrottleTests.cs && git commit -m "fix(security): amortize LoginThrottle.Prune off the failed-bind hot path (plan R2-07 T11)"`
|
||||
|
||||
### Task 12: Lock in the scrubber's fragment coverage + document the array-merge limitation
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1, 2, 4, 5, 7, 10, 11
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ConfigSecretScrubber.cs` (`IsSecretName` :21 → `internal`; `MergeSentinels` xmldoc :121-126)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/ConfigSecretScrubberTests.cs` (extend)
|
||||
|
||||
Detection is name-fragment-based (`password|secret|token|apikey|credential|passphrase`, :18-19). Current config types are fully covered, but a future field named `SigningKey` or `Pin` would silently leak (N9). Freeze the coverage the same way `RequiredRoleMatrixTests` froze the authz table: enumerate every string property on the DataConnections config types and demand each is *either* fragment-matched (scrubbed) *or* on a deliberate non-secret allowlist.
|
||||
|
||||
1. Change `IsSecretName` from `private` to `internal` (the csproj already has `InternalsVisibleTo` for the test project). Write the failing test:
|
||||
```csharp
|
||||
using System.Reflection;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.DataConnections;
|
||||
|
||||
// Frozen non-secret allowlist (arch-review R2 N9), mirroring RequiredRoleMatrixTests:
|
||||
// every string-typed property on the DataConnections config types must EITHER match
|
||||
// the scrubber's secret fragments (it gets elided) OR be listed here after a
|
||||
// deliberate "this is not a secret" decision. A new string property with neither
|
||||
// fails EveryConfigStringProperty_IsScrubbedOrDeliberatelyNonSecret, so a future
|
||||
// "SigningKey"/"Pin" cannot silently leak through List/Get/audit.
|
||||
private static readonly Dictionary<Type, string[]> KnownNonSecretStringProperties = new()
|
||||
{
|
||||
[typeof(OpcUaEndpointConfig)] = ["EndpointUrl", "SubscriptionDisplayName"],
|
||||
[typeof(OpcUaUserIdentityConfig)] = ["Username", "CertificatePath"],
|
||||
[typeof(MxGatewayEndpointConfig)] = ["Endpoint", "ClientName", "CaFile", "ServerName"],
|
||||
[typeof(OpcUaHeartbeatConfig)] = [/* enumerate on first run; likely none */],
|
||||
[typeof(OpcUaDeadbandConfig)] = [/* likely none */],
|
||||
};
|
||||
|
||||
[Fact]
|
||||
public void EveryConfigStringProperty_IsScrubbedOrDeliberatelyNonSecret()
|
||||
{
|
||||
foreach (var (type, allowlist) in KnownNonSecretStringProperties)
|
||||
{
|
||||
var stringProps = type.GetProperties(BindingFlags.Public | BindingFlags.Instance)
|
||||
.Where(p => p.PropertyType == typeof(string))
|
||||
.Select(p => p.Name);
|
||||
foreach (var name in stringProps)
|
||||
{
|
||||
var scrubbed = ConfigSecretScrubber.IsSecretName(name);
|
||||
var allowlisted = allowlist.Contains(name);
|
||||
Assert.True(scrubbed ^ allowlisted,
|
||||
$"{type.Name}.{name}: must be EITHER fragment-scrubbed OR explicitly " +
|
||||
"allowlisted as non-secret (and never both) — see arch-review R2 N9.");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData("Password")]
|
||||
[InlineData("CertificatePassword")]
|
||||
[InlineData("ApiKey")]
|
||||
public void KnownSecretProperties_MatchTheFragmentList(string name)
|
||||
=> Assert.True(ConfigSecretScrubber.IsSecretName(name));
|
||||
```
|
||||
Cross-check the allowlist against the actual types before committing (`src/ZB.MOM.WW.ScadaBridge.Commons/Types/DataConnections/` — `OpcUaEndpointConfig.cs`, `OpcUaUserIdentityConfig.cs`, `MxGatewayEndpointConfig.cs`, `OpcUaHeartbeatConfig.cs`, `OpcUaDeadbandConfig.cs`); the XOR also locks the *secret* side (a rename of `Password` off the fragment list fails the test).
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests --filter ConfigSecretScrubber` — FAIL first on compile (`IsSecretName` private), then green the assertions.
|
||||
3. Append to the `MergeSentinels` xmldoc (:121-126) the N9 limitation, so the by-index behavior is a documented contract rather than a surprise:
|
||||
```
|
||||
/// <b>Array limitation (arch-review R2 N9):</b> sentinel restoration inside arrays
|
||||
/// pairs incoming[i] with stored[i] BY INDEX. A client that reorders an array of
|
||||
/// credentialed objects while round-tripping sentinels grafts the wrong stored
|
||||
/// secret into each slot — a silent misconfiguration (never a disclosure). Clients
|
||||
/// must not reorder arrays that carry sentinels (the CLI round-trip does not);
|
||||
/// key-based matching is a possible future refinement if a config type ever nests
|
||||
/// credentialed arrays.
|
||||
```
|
||||
4. Run the filter — PASS; full `dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests` run.
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ManagementService/ConfigSecretScrubber.cs tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/ConfigSecretScrubberTests.cs && git commit -m "test(management): freeze scrubber fragment coverage over DataConnection config types + document array-merge limit (plan R2-07 T12)"`
|
||||
|
||||
---
|
||||
|
||||
## Dependencies on other plans
|
||||
|
||||
- **PLAN-R2-02 (Communication / site stream):** shares `SiteAlarmLiveCacheService.cs` — **Task 10 and R2-02's delta-coalescing task are mutually exclusive in time** (see the mutex rule at the top). The gRPC/site half of the live-alarm path (stream reconnect, `SubscribeSite` semantics) belongs to R2-02; Task 10 touches only the central service's liveness bookkeeping.
|
||||
- **ManagementActor single-writer lane:** only this plan (Tasks 4, 6) touches `ManagementActor.cs` in round 2, but the initiative-wide rule stands — serialize against any other plan's ManagementActor task that may appear.
|
||||
- **Round-1 documented deferrals stay put:** S1's client-supplied import-id idempotency remains plan-05 domain; S4 streaming multipart, P2 QueryDeployments/ExportBundle projections, and the UA6 fleet-wide grid a11y sweep remain in `docs/plans/2026-07-08-deferred-work-register.md`. No round-2 task may "helpfully" absorb them.
|
||||
|
||||
## Execution order
|
||||
|
||||
**P0 (security-correctness first):** Task 1 (N1) and Task 2 → 3 (N2) and Task 4 → 6 (N3, with Task 5 landing any time before 6) — these five are the Medium security findings. Then Task 7 (N4, the operator-facing Medium) → 8 → 9 on the AlarmSummary lane; Tasks 10, 11, 12 land any time (10 subject to the R2-02 mutex).
|
||||
|
||||
**Ordering constraints:** 3←2; 6←(4,5); 8←7; 9←8. Serialize {4, 6} (ManagementActor lane) and {7, 8, 9} (AlarmSummary.razor). Everything else is file-disjoint.
|
||||
|
||||
**Milestone verification:** at plan completion run `dotnet build ZB.MOM.WW.ScadaBridge.slnx` and `dotnet test ZB.MOM.WW.ScadaBridge.slnx` (targeted filters during tasks; the full sweep only here), then rebuild the docker cluster (`bash docker/deploy.sh`) for a smoke: a wrong-password CLI login 401s then 429s on the 6th try (throttle through Traefik with the real client IP), `secured-write list` under `multi-role` still works, and the Alarm Summary page live-updates then survives a site switch mid-poll.
|
||||
|
||||
---
|
||||
|
||||
## Findings Coverage
|
||||
|
||||
| Finding | Severity | Disposition |
|
||||
|---|---|---|
|
||||
| N1 — DebugStreamHub binds LDAP with no throttle; `Component-Security.md:208` falsely claims complete coverage | Medium | Task 1 (hub delegates to `ManagementAuthenticator.AuthenticateAsync`; doc corrected in the same task) |
|
||||
| N2 — LoginThrottle keys collapse behind Traefik: no ForwardedHeaders handling anywhere | Medium | Tasks 2, 3 (trusted-proxy `UseForwardedHeaders` bound from config; docker/docker-env2/deploy config shipped; lockout-DoS trade-off + mitigation documented; throttle-test assumption noted) |
|
||||
| N3 — Secured-write handlers ignore site scope while the spec says scoping is layered on at the LDAP-mapping level | Medium | Tasks 4, 5, 6 (enforce at submit/approve/reject via existing `EnforceSiteScope*` helpers; scoped list at the query level; `Component-Security.md:141` amended to match; frozen matrix verified — no role changes, no regeneration needed) |
|
||||
| N4 — `RefreshAsync` applies fan-out results without re-checking the selected site (cross-site data after a switch) | Medium | Task 7 (one-line guard mirroring the live path's `:374` check + bUnit regression) |
|
||||
| N5 — When live, a slower poll snapshot can regress fresher live state | Low | Task 8 (poll updates only `_notReporting` while `IsLive`; stale comment fixed) |
|
||||
| N6 — `IsLive` is sticky: `HasPublished` never resets on aggregator death | Low | Task 10 (deathwatch → liveness reset + snapshot clear + self-heal restart; interface doc updated). **Mutex with PLAN-R2-02's coalescing task.** |
|
||||
| N7 — Live-callback `InvokeAsync` lacks the house disposal guard | Low | Task 9 (`_disposed` set before teardown, checked before and inside the marshal — DebugView pattern) |
|
||||
| N8 — `LoginThrottle.Prune` scans (and may sort) the whole map on every failed bind | Low | Task 11 (every-Nth-failure cadence + immediate over-cap sweep; semantics unchanged) |
|
||||
| N9 — Scrubber array merge is by-index; fragment list has no lock-in against future config fields | Low | Task 12 (frozen XOR coverage test over the DataConnections config types; array-by-index limitation documented as contract) |
|
||||
| *Round-1 residue* S1 — import idempotency on a client-supplied id | High (residual) | **Remains deferred** (round-1 documented deferral; plan-05 / Transport apply-path domain; tracked in the deferred-work register) |
|
||||
| *Round-1 residue* S4 — streaming multipart bundle upload | Medium (residual) | **Remains deferred** (bounded by the round-1 `TransportGate` single-flight; deferred-work register) |
|
||||
| *Round-1 residue* P2 — QueryDeployments/ExportBundle load-all internals | Medium (residual) | **Remains deferred** (bounded by fleet size; needs repo-level projections; deferred-work register) |
|
||||
| *Round-1 residue* UA6 — fleet-wide sortable-grid a11y sweep | UA (residual) | **Remains deferred** (uniform low-severity UX debt; logged in the AlarmSummary component comment + deferred-work register) |
|
||||
| *Round-1 residue* P1/UA5 — the one missed bind surface | Medium (residual) | **Owned here as N1** → Task 1 (no separate row-work) |
|
||||
@@ -0,0 +1,87 @@
|
||||
{
|
||||
"planPath": "archreview/plans/PLAN-R2-07-ui-management-security.md",
|
||||
"tasks": [
|
||||
{
|
||||
"id": 1,
|
||||
"subject": "Task 1: Route the DebugStreamHub LDAP bind through ManagementAuthenticator (throttled) + fix the false doc claim",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"subject": "Task 2: ForwardedHeadersSetup — trusted-proxy client-IP resolution for the throttle keys",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"subject": "Task 3: Ship the ForwardedHeaders config to the Traefik topologies + document the lockout-DoS trade-off",
|
||||
"status": "pending",
|
||||
"blockedBy": [
|
||||
2
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"subject": "Task 4: Enforce site scope on secured-write submit / approve / reject",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"subject": "Task 5: ISecuredWriteRepository — additive permitted-sites filter for scoped listing",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"subject": "Task 6: Scope-filter HandleListSecuredWrites + amend Component-Security.md:141 + matrix verification",
|
||||
"status": "pending",
|
||||
"blockedBy": [
|
||||
4,
|
||||
5
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
"subject": "Task 7: AlarmSummary.RefreshAsync stale-site guard — never display cross-site data",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": 8,
|
||||
"subject": "Task 8: While live, the poll updates only _notReporting — never regresses live rows",
|
||||
"status": "pending",
|
||||
"blockedBy": [
|
||||
7
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 9,
|
||||
"subject": "Task 9: House disposal guard on the live callback",
|
||||
"status": "pending",
|
||||
"blockedBy": [
|
||||
8
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 10,
|
||||
"subject": "Task 10: Un-stick IsLive — deathwatch resets liveness when the aggregator terminates (MUTEX with PLAN-R2-02 SiteAlarmLiveCacheService task)",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": 11,
|
||||
"subject": "Task 11: Amortize LoginThrottle.Prune off the failure hot path",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": 12,
|
||||
"subject": "Task 12: Lock in the scrubber's fragment coverage + document the array-merge limitation",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
}
|
||||
],
|
||||
"lastUpdated": "2026-07-13T03:26:45Z"
|
||||
}
|
||||
@@ -0,0 +1,504 @@
|
||||
# PLAN-R2-08 — Conventions & Hygiene (Round-2 Findings NF1–NF8)
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
|
||||
|
||||
**Goal:** Close the eight NEW findings from architecture review 08 round 2 (`archreview/08-conventions-tests-underdeveloped.md`, 2026-07-12) — rotate the live production Inbound API key sitting in `test.txt` at the repo root and purge the other plaintext credential captures (NF1, High), refresh the abandoned-and-now-wrong CHANGELOG (NF3), finish the options-validation tail (AuditLog's 6 sub-options + Host Node/Database/Logging + OperationTracking — NF4), re-consolidate deferral tracking into the canonical register (NF5), fix the stale Transport area comment (NF6), commit the orphaned MES plan doc + clean root clutter (NF7), and kill the duplicate/drifted Host options bindings (NF8). NF2 (failover envelope measurement) is **owned by PLAN-R2-01** — this plan only repairs the register row whose revisit trigger fired.
|
||||
|
||||
**Architecture:** No new runtime components, no schema changes, no wire-contract changes. The options work copies the shipped `OptionsValidatorBase<T>` + `AddValidatedOptions<T, TValidator>` pattern (`ZB.MOM.WW.Configuration`; see `AuditLog/ServiceCollectionExtensions.cs:81` and `KpiHistory/ServiceCollectionExtensions.cs:40`). The credential work is operational (dual-key rotation per `docs/requirements/Component-InboundAPI.md` §Key rotation, shipped by PLAN-06 T24) plus working-tree hygiene with a `.gitignore`/pre-commit guard. One discovery made while verifying NF4 is folded in as Task 8: the site-only `IOperationTrackingStore` appears to have **no DI registration anywhere in `src/`**, despite two comments claiming `AddSiteRuntime` registers it — Task 8 verifies and fixes that seam alongside the missing `OperationTrackingOptions` binding.
|
||||
|
||||
**Tech Stack:** C# / .NET 10, xUnit, `Microsoft.Extensions.Options` (`IValidateOptions<>` + `ValidateOnStart`), `ZB.MOM.WW.Configuration` (`OptionsValidatorBase<T>`, `AddValidatedOptions`), SQLite (site tracking store), ScadaBridge CLI (`security api-key` group), git/bash (hygiene tasks).
|
||||
|
||||
**Build:** `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Tests: `dotnet test ZB.MOM.WW.ScadaBridge.slnx` (all 30 test projects, incl. CLI.Tests).
|
||||
|
||||
**Baseline:** `main` @ `8c888f13` (2026-07-12). Re-verify cited line numbers before editing — they were checked at plan-authoring time against this commit.
|
||||
|
||||
## Parallelization
|
||||
|
||||
- **Task 1 FIRST** (NF1 is the P0; it needs the user/host, so start it immediately and let the wait overlap other tasks). Task 2 follows Task 1 (rotate before destroying the local record of what was exposed).
|
||||
- **Mutually parallel (file-disjoint):** 1, 3, 4, 5, 7, 10, 12, 13.
|
||||
- **Serial chains (shared files):** 5 → 6 (both edit `AuditLog/ServiceCollectionExtensions.cs`); 7 → 8 → 9 (all edit `Host/SiteServiceRegistration.cs`); 10 → 11 (delete the snapshot only after the register has absorbed its content).
|
||||
- Tasks 1, 2, 11, and the deletion half of 13 are flagged **`needs-user`** — they require production-host access and/or destroy untracked (unrecoverable) files, so the executor must show the user what is being done and get confirmation.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Rotate the exposed wonder-app-vd03 Inbound API key (dual-key flow) — `needs-user`
|
||||
|
||||
**Classification:** NF1 — high-risk security hygiene; **`needs-user`** (production environment access; the executor runs the commands only with the user present/consenting, and cannot complete step 3's caller-cutover without the user)
|
||||
**Estimated implement time:** ~5 min of active work (cutover/quarantine waits excluded)
|
||||
**Parallelizable with:** 3, 4, 5, 7, 10, 12, 13
|
||||
**Files:**
|
||||
- Modify: `docs/deployment/deployment-records/2026-06-27-wonder-app-vd03.md` (append a dated rotation record — the only repo artifact this task produces)
|
||||
|
||||
**Context (verified):** `test.txt` (untracked, repo root) contains a **working production Inbound API key** for `http://wonder-app-vd03.zmr.zimmer.com:8085` — `sbk_eb5acc0b…` (full token on line 3) scoped at least to `MesMoveOut`. The key is live regardless of whether the file ever enters git history, so it must be **rotated**, not merely deleted. The zero-downtime dual-key procedure is already documented (`docs/requirements/Component-InboundAPI.md:75-86`, PLAN-06 T24): create → cutover → disable → quarantine → delete. This task is that procedure instantiated against wonder-app-vd03 as an executable checklist. Host facts (auto-memory `wonder-app-vd03-host-access`): the deployment runs with `DisableLogin=true`, so `:8085/management` is unauthenticated and the CLI works with just `--url`; host shell access (if needed) is servecli SSH port 2222.
|
||||
|
||||
**Checklist (run each step, record output):**
|
||||
|
||||
1. Enumerate keys and identify the exposed one (match the `eb5acc0b` key-id prefix from `test.txt`; note its **name, key-id, and scope list**):
|
||||
```bash
|
||||
scadabridge --url http://wonder-app-vd03.zmr.zimmer.com:8085 security api-key list
|
||||
```
|
||||
2. Create the replacement with the **same scope set** (copy the old key's method list verbatim from step 1 — scopes are copied at creation only; drift between the pair is operator error):
|
||||
```bash
|
||||
scadabridge --url http://wonder-app-vd03.zmr.zimmer.com:8085 security api-key create \
|
||||
--name "<old-name>-rot-2026-07" --methods "<exact scope list from step 1>"
|
||||
```
|
||||
Capture the one-time `sbk_…` token from the output immediately (it is shown once). Do **not** write it to any file in this repo — hand it to the user directly in the conversation or via their chosen secret channel.
|
||||
3. **USER ACTION — cutover:** the user delivers the new token to the calling system's owner (the MES/DELMIA-side integration that drives `MesMoveOut`) and confirms the caller has switched. Verify cutover from the audit log (the Actor field records the serving key per call): `scadabridge --url http://wonder-app-vd03.zmr.zimmer.com:8085 audit list --channel InboundApi --limit 20` (adjust flags to the real `audit` verbs) — recent `MesMoveOut` rows must show the **new** key id.
|
||||
4. Disable (don't delete) the old key once cutover is confirmed:
|
||||
```bash
|
||||
scadabridge --url http://wonder-app-vd03.zmr.zimmer.com:8085 security api-key update --key-id <old-key-id> --enabled false
|
||||
```
|
||||
5. Schedule deletion after a quarantine window (user judgment; suggest ≥ 1 week):
|
||||
```bash
|
||||
scadabridge --url http://wonder-app-vd03.zmr.zimmer.com:8085 security api-key delete --key-id <old-key-id>
|
||||
```
|
||||
6. Append a dated "API-key rotation 2026-07" subsection to `docs/deployment/deployment-records/2026-06-27-wonder-app-vd03.md`: why (key exposed in a local scratch file, arch-review 08 round 2 NF1), old key id (id only — never the secret), new key name/id, cutover-confirmed date, disable date, planned delete date. Do NOT include any token material.
|
||||
7. Verify (hygiene task — verification replaces a test): step 1 re-run shows the old key `Enabled=false` (or absent after step 5) and the new key present; `grep -c "sbk_" docs/deployment/deployment-records/2026-06-27-wonder-app-vd03.md` returns `0`.
|
||||
8. Commit:
|
||||
```bash
|
||||
git add docs/deployment/deployment-records/2026-06-27-wonder-app-vd03.md
|
||||
git commit -m "chore(hygiene): record wonder-app-vd03 inbound API-key rotation — exposed key retired via dual-key flow (plan R2-08 T1)"
|
||||
```
|
||||
|
||||
If the user cannot reach wonder-app-vd03 in this session, record the checklist state (which steps are pending) in the deployment record with a `PENDING` marker and still commit — the rotation must not be silently forgotten. Task 2 may proceed regardless (deleting `test.txt` does not need the old secret; disable/delete key operations only need the key-id from `list`).
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Delete `test.txt` and triage the root credential files — `needs-user`
|
||||
|
||||
**Classification:** NF1 — high-risk security hygiene; **`needs-user`** (deletes untracked files — unrecoverable; user must see and confirm each)
|
||||
**Estimated implement time:** ~3 min
|
||||
**Parallelizable with:** 3, 4, 5, 7, 10, 12, 13 (blocked by 1 — rotate before destroying the local record; see Task 1's closing note for the pending-rotation escape hatch)
|
||||
**Files:**
|
||||
- Delete (untracked): `test.txt`
|
||||
- Delete (untracked+ignored, user-confirmed): `ldap_login.txt`, `sql_login.txt`, `sqllogin.txt`
|
||||
|
||||
**Context (verified by reading each file):**
|
||||
- `test.txt` — the live `sbk_eb5acc0b…` production API key (Task 1's subject) plus an ad-hoc test note. Untracked and NOT gitignored — one `git add -A` (or the `pushit` skill, which stages everything) from entering history.
|
||||
- `ldap_login.txt` — **live AD credentials**: `zimdom.zimmer.com:3268` bind with the `SVC-DNC` service-account password AND the user's personal domain password. Ignored via `.gitignore:45` (`*login*.txt`) so unstageable, but plaintext on disk.
|
||||
- `sql_login.txt` — the wonder-app-vd03 SQL Server `wwadmin` password (the production ScadaBridge config DB). Ignored, plaintext.
|
||||
- `sqllogin.txt` — a provisioning transcript that creates `wwadmin` **with sysadmin** using the same password. Ignored, plaintext.
|
||||
|
||||
None of the four are in git history (verified: untracked/ignored; `git log` shows no commits touching them), so this is disk cleanup, not history surgery.
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Show the user the four files' contents (Read each aloud in the conversation) and state plainly: they are untracked — deletion is unrecoverable. Get explicit confirmation per file. If the user wants any kept, move it OUTSIDE the repo (e.g. `~/secrets/`) instead of leaving it in the working tree.
|
||||
2. On confirmation:
|
||||
```bash
|
||||
rm /Users/dohertj2/Desktop/ScadaBridge/test.txt
|
||||
rm /Users/dohertj2/Desktop/ScadaBridge/ldap_login.txt /Users/dohertj2/Desktop/ScadaBridge/sql_login.txt /Users/dohertj2/Desktop/ScadaBridge/sqllogin.txt
|
||||
```
|
||||
3. **Advisory to relay to the user (do not action without them):** the SVC-DNC, personal-domain, and `wwadmin` passwords in those files were exposed on disk for ~7 weeks; recommend rotating them through the normal AD/SQL channels. The `wwadmin` password also appears in the on-host deployment; rotating it means updating `appsettings` connection strings on wonder-app-vd03 (`deploy/wonder-app-vd03/RUNBOOK.md` procedure).
|
||||
4. Verify: `ls test.txt ldap_login.txt sql_login.txt sqllogin.txt 2>&1` — all four report "No such file". `git status --short` no longer lists `test.txt`.
|
||||
5. No commit — the deletions touch only untracked files (Task 3 commits the guard that prevents recurrence).
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Root secret-capture guards — `.gitignore` patterns + pre-commit secret scan
|
||||
|
||||
**Classification:** small (hygiene guard)
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12
|
||||
**Files:**
|
||||
- Modify: `.gitignore` (currently 48 lines; the `*login*.txt` rule is at line 45)
|
||||
- Create: `tools/hooks/pre-commit` (secret-scan hook) and `tools/hooks/README.md` (2-line install note)
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Extend `.gitignore` — after the existing `*login*.txt` block (line 45), add:
|
||||
```gitignore
|
||||
# Ad-hoc scratch/credential captures at the repo root — never commit
|
||||
# (arch-review 08 round 2 NF1: a live API key sat in an untracked test.txt)
|
||||
/test.txt
|
||||
/*credentials*
|
||||
/*secret*
|
||||
|
||||
# Claude tool worktrees (NF7)
|
||||
.claire/
|
||||
```
|
||||
Keep patterns root-anchored (`/`) so legitimate `src/`/`tests/` files named e.g. `SecretsBlock.cs` are unaffected.
|
||||
2. Create `tools/hooks/pre-commit` (mode 755) — a ~20-line POSIX-sh scan of the **staged diff** that blocks the commit when it introduces obvious credential material:
|
||||
- an inbound API token: `sbk_[0-9a-f]{16,}_[A-Za-z0-9_-]{20,}`
|
||||
- a plaintext password assignment in root-level `.txt`/`.md` additions: `^\+.*[Pp]assword[0-9]*\s*[:=]\s*\S`
|
||||
Exit non-zero with a message naming the offending file/pattern and how to bypass (`git commit --no-verify`) for a knowing false positive. Scan only `git diff --cached -U0` output — never the whole tree (speed + no false hits on committed test fixtures).
|
||||
3. `tools/hooks/README.md`: two lines — what the hook does, and the opt-in install command `git config core.hooksPath tools/hooks` (hooks are not tracked by git; this is a suggestion the user enables once, exactly the "pre-commit suggestion" the report asked for).
|
||||
4. Verify (guard task — a real fail→pass exercise replaces a unit test):
|
||||
```bash
|
||||
git check-ignore -v test.txt .claire/ # both match the new rules
|
||||
printf 'api key: sbk_%s_%s\n' "$(printf 'a%.0s' {1..32})" "$(printf 'b%.0s' {1..40})" > /tmp/nf1-probe.txt
|
||||
cp /tmp/nf1-probe.txt probe-secret-scan.md && git add probe-secret-scan.md
|
||||
sh tools/hooks/pre-commit; echo "hook exit: $?" # expect non-zero (blocked)
|
||||
git reset probe-secret-scan.md && rm probe-secret-scan.md /tmp/nf1-probe.txt
|
||||
sh tools/hooks/pre-commit; echo "hook exit: $?" # clean index → expect 0
|
||||
```
|
||||
5. Commit:
|
||||
```bash
|
||||
git add .gitignore tools/hooks
|
||||
git commit -m "chore(hygiene): root secret-capture guards — .gitignore patterns + opt-in pre-commit secret scan (plan R2-08 T3)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 4: CHANGELOG.md — correct the false role claim and refresh to reality
|
||||
|
||||
**Classification:** trivial (documentation)
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13
|
||||
**Files:**
|
||||
- Modify: `CHANGELOG.md` (last touched 2026-06-02, `b104760b`; the false claim is at lines 11-12)
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Fix the falsehood first — lines 11-12 assert "`Operator`/`Engineer` are unused by ScadaBridge", false since M7 shipped the secured-write flow. Replace the parenthetical with:
|
||||
> (`Engineer` remains unused by ScadaBridge; `Operator` has since been adopted by the M7 two-person secured-write flow, together with the new `Verifier` role — see the M7 entry above)
|
||||
2. Add new entries ABOVE the existing `[Unreleased]` content (keep-a-changelog style, newest first, still under `[Unreleased]` since nothing is version-tagged). Keep it honest and SHORT — 3 summary blocks, a few bullets each, each pointing at the authoritative record rather than duplicating it:
|
||||
- **2026-07-08 → 2026-07-10 — Architecture-review hardening initiative (PLAN-01…PLAN-08).** One paragraph: 8 reviews → 8 fix plans, 191/192 tasks landed (options validation across 12+ components, ClusterClient contract-lock tests, secured-write TTL, login throttle, transport single-flight, S&F sweep bounding, deploy-readiness handshake, vestigial site-notification surface excised, deferred-work register established). Authoritative record: `archreview/plans/00-MASTER-TRACKER.md` + the per-plan docs.
|
||||
- **M8 — Transport.** Encrypted bundle export/import extended to site/instance-scoped config with `BundleNameMap` reconciliation, per-line Myers diff, script trust gate at import. Post-initiative: aggregated live alarm stream + KPI hourly rollups (both 2026-07-10).
|
||||
- **M7 — Native alarms + operational UX.** Native alarm mirroring (OPC UA A&C + MxGateway), Alarm Summary page, OPC UA browse/search/cert-trust, **two-person secured writes introducing the `Operator`/`Verifier` roles**, SMS notification adapter.
|
||||
3. Add one closing line under the changelog intro: detailed per-change history lives in `git log`, `archreview/plans/00-MASTER-TRACKER.md`, and `docs/plans/` — this file records operator-facing/breaking changes only.
|
||||
4. Verify: `grep -n "Engineer remains unused\|Verifier" CHANGELOG.md` hits; `grep -n "are unused by ScadaBridge" CHANGELOG.md` → no hit for the old combined claim; the two BREAKING sections (roles, inbound-API auth) are preserved untouched below the new entries.
|
||||
5. Commit:
|
||||
```bash
|
||||
git add CHANGELOG.md
|
||||
git commit -m "docs(changelog): refresh to reality — fix false Operator/Engineer claim, summarize arch-review initiative + M7/M8 (plan R2-08 T4)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 5: Options validation — AuditLog site sub-options (SiteWriter, SiteTelemetry, SiteRetention)
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1, 2, 3, 4, 7, 10, 11, 12, 13 (NOT 6 — both edit `AuditLog/ServiceCollectionExtensions.cs`)
|
||||
**Files:**
|
||||
- Create: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriterOptionsValidator.cs`, `src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/SiteAuditTelemetryOptionsValidator.cs`, `src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SiteAuditRetentionOptionsValidator.cs`
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/ServiceCollectionExtensions.cs` (lines 100-108 — the three `AddOptions<T>().Bind(...)` calls)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SqliteAuditWriterOptionsValidatorTests.cs`, `.../Site/SiteAuditTelemetryOptionsValidatorTests.cs`, `.../Site/SiteAuditRetentionOptionsValidatorTests.cs`
|
||||
|
||||
The pattern to copy is in the SAME file: `AuditLog/ServiceCollectionExtensions.cs:81` already does `services.AddValidatedOptions<AuditLogOptions, AuditLogOptionsValidator>(config, ConfigSectionName)` — the shared helper (scadaproj `ZB.MOM.WW.Configuration/ServiceCollectionExtensions.cs:28-41`) binds via `config.GetSection(sectionPath)` (preserving the file's documented partial-config-root rationale at lines 96-99), chains `.ValidateOnStart()`, and `TryAddEnumerable`s the validator. Validator + test shape: `AuditLog/Configuration/AuditLogOptionsValidator.cs` / `AuditLog.Tests/Configuration/AuditLogOptionsValidatorTests.cs`.
|
||||
|
||||
1. Write failing tests first (one `DefaultOptions_AreValid` + 2-3 rejection cases per validator, `new XValidator().Validate(Options.DefaultName, options)` shape). Rejection cases (fields verified against the option classes):
|
||||
- `SqliteAuditWriterOptions`: `DatabasePath = ""`, `ChannelCapacity = 0`, `BatchSize = 0`, `FlushIntervalMs = 0`. **Do NOT validate `BacklogPollIntervalSeconds`** — non-positive has a documented fall-back-to-30s contract (register row 21 resolution + cadence tests); note this in a validator comment.
|
||||
- `SiteAuditTelemetryOptions`: `BatchSize = 0`, `BusyIntervalSeconds = 0`, `IdleIntervalSeconds = 0`. Read `SiteAuditTelemetryActor`'s tick usage first; if idle < busy is genuinely nonsensical there, add the `IdleIntervalSeconds >= BusyIntervalSeconds` cross-check too.
|
||||
- `SiteAuditRetentionOptions`: `RetentionDays = 0`, `PurgeInterval = TimeSpan.Zero`, `InitialDelay = TimeSpan.FromSeconds(-1)` (require `>= TimeSpan.Zero`). Leave `PurgeIntervalOverride` unvalidated — the class remarks mark it test-only, not config-set.
|
||||
2. Run, expect FAIL (compile error on missing validator types counts):
|
||||
```bash
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter "FullyQualifiedName~SqliteAuditWriterOptionsValidator"
|
||||
```
|
||||
3. Implement the three validators (`OptionsValidatorBase<T>`, key-naming messages using the real section names from the SCE constants — `AuditLog:SiteWriter`, `AuditLog:SiteTelemetry`, `AuditLog:SiteRetention`). Convert the bindings at lines 100-108 to:
|
||||
```csharp
|
||||
services.AddValidatedOptions<SqliteAuditWriterOptions, SqliteAuditWriterOptionsValidator>(config, SiteWriterSectionName);
|
||||
services.AddValidatedOptions<SiteAuditTelemetryOptions, SiteAuditTelemetryOptionsValidator>(config, SiteTelemetrySectionName);
|
||||
services.AddValidatedOptions<SiteAuditRetentionOptions, SiteAuditRetentionOptionsValidator>(config, SiteAuditRetentionOptions.SectionName);
|
||||
```
|
||||
Keep (condense, don't delete) the surrounding why-comments — the partial-config-root note and the "inert on central" retention note still hold.
|
||||
4. Run, expect PASS; then guard both composition roots (central also calls `AddAuditLog`, so defaults must validate clean there):
|
||||
```bash
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter "FullyQualifiedName~OptionsValidator"
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~CompositionRoot"
|
||||
```
|
||||
5. Commit:
|
||||
```bash
|
||||
git add src/ZB.MOM.WW.ScadaBridge.AuditLog tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests
|
||||
git commit -m "fix(options): eager startup validation for AuditLog site sub-options — SiteWriter, SiteTelemetry, SiteRetention (plan R2-08 T5, arch-review 08r2 NF4)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 6: Options validation — AuditLog central sub-options (PartitionMaintenance, Purge, Reconciliation)
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1, 2, 3, 4, 7, 10, 11, 12, 13 (blocked by 5 — same `ServiceCollectionExtensions.cs`)
|
||||
**Files:**
|
||||
- Create: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPartitionMaintenanceOptionsValidator.cs`, `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPurgeOptionsValidator.cs`, `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/SiteAuditReconciliationOptionsValidator.cs`
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/ServiceCollectionExtensions.cs` (lines 368-390 in `AddAuditLogCentralMaintenance` — re-check offsets after Task 5's edit)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Central/AuditLogPartitionMaintenanceOptionsValidatorTests.cs`, `.../Central/AuditLogPurgeOptionsValidatorTests.cs`, `.../Central/SiteAuditReconciliationOptionsValidatorTests.cs`
|
||||
|
||||
Same recipe as Task 5. Rejection cases (fields verified):
|
||||
- `AuditLogPartitionMaintenanceOptions` (`AuditLog:PartitionMaintenance`): `IntervalSeconds = 0`, `LookaheadMonths = 0`.
|
||||
- `AuditLogPurgeOptions` (`AuditLog:Purge`): `IntervalHours = 0`, `ChannelPurgeBatchSizeConfigured = 0`, `MaintenanceCommandTimeoutMinutes = 0`. Leave `IntervalOverride` unvalidated (test-only per class remarks).
|
||||
- `SiteAuditReconciliationOptions` (`AuditLog:Reconciliation`): `ReconciliationIntervalSeconds = 0`, `BatchSize = 0`, `StalledAfterNonDrainingCycles = 0`. Leave `ReconciliationIntervalOverride` unvalidated.
|
||||
|
||||
1. Failing tests → 2. run one filter, expect FAIL → 3. implement validators + convert the three `AddOptions<T>().Bind(...)` at :368-390 to `AddValidatedOptions` with the existing section constants (`PartitionMaintenanceSectionName`, `PurgeSectionName`, `ReconciliationSectionName`); **keep the long I1-review comment block at :372-384** — it explains WHY the two singleton options bind here, which is unchanged → 4. run, expect PASS:
|
||||
```bash
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter "FullyQualifiedName~OptionsValidator"
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~CompositionRoot"
|
||||
```
|
||||
2. Commit:
|
||||
```bash
|
||||
git add src/ZB.MOM.WW.ScadaBridge.AuditLog tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests
|
||||
git commit -m "fix(options): eager startup validation for AuditLog central sub-options — PartitionMaintenance, Purge, Reconciliation (plan R2-08 T6, arch-review 08r2 NF4)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 7: Options validation — Host NodeOptions / DatabaseOptions / LoggingOptions (empty NodeName fails fast)
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1, 2, 3, 4, 5, 6, 10, 11, 12, 13 (NOT 8/9 — all three edit `Host/SiteServiceRegistration.cs`)
|
||||
**Files:**
|
||||
- Create: `src/ZB.MOM.WW.ScadaBridge.Host/NodeOptionsValidator.cs`, `src/ZB.MOM.WW.ScadaBridge.Host/DatabaseOptionsValidator.cs`, `src/ZB.MOM.WW.ScadaBridge.Host/LoggingOptionsValidator.cs`
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs` (`BindSharedOptions`, lines 160/165/172 — the plain `services.Configure<>` calls for these three)
|
||||
- Modify tests: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CompositionRootTests.cs` (in-memory config dicts at ~:111 and ~:350), `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/MetricsEndpointTests.cs` (~:43) — none currently sets `ScadaBridge:Node:NodeName`
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/NodeOptionsValidatorTests.cs` (one file covering all three validators is fine)
|
||||
|
||||
This is the SourceNode failure mode the tracker logged for wonder-app-vd03: an empty `NodeOptions.NodeName` "normalises to a NULL SourceNode" (the appsettings `_nodeName` comment says so verbatim) — it must fail the host at boot, not silently degrade the audit trail. The Host csproj already references `ZB.MOM.WW.Configuration` transitively via components, but add the explicit `<PackageReference Include="ZB.MOM.WW.Configuration" />` (no Version — central package management) if `dotnet build` says otherwise.
|
||||
|
||||
1. Write failing validator tests. Rejection cases:
|
||||
- `NodeOptions`: `NodeName = ""` (and whitespace) rejected — message must say what breaks ("SourceNode audit column stamps NULL"). `RemotingPort`/`GrpcPort`/`MetricsPort` outside `0..65535` rejected (**0 must stay VALID** — `CompositionRootTests` uses `RemotingPort = 0` for dynamic ports).
|
||||
- `DatabaseOptions`: all three fields are nullable and role-dependent — reject only whitespace-non-null values (e.g. `ConfigurationDb = " "`); null stays valid.
|
||||
- `LoggingOptions`: `MinimumLevel = "Chatty"` rejected. First read how `MinimumLevel` is consumed (Program.cs Serilog bootstrap) and validate against that parser's accepted set (Serilog: Verbose/Debug/Information/Warning/Error/Fatal, case-insensitive).
|
||||
2. Run, expect FAIL:
|
||||
```bash
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~NodeOptionsValidator"
|
||||
```
|
||||
3. Implement the three validators (`OptionsValidatorBase<T>`, key prefix `ScadaBridge:Node` / `ScadaBridge:Database` / `ScadaBridge:Logging`). Convert `BindSharedOptions` lines 160/165/172 to the `AddOptions<T>().Bind(...).ValidateOnStart()` + `TryAddEnumerable` shape (exactly what the same method already does for `ClusterOptions` at :164 and `HealthMonitoringOptions` at :170 — mirror those, comments included).
|
||||
4. Add `["ScadaBridge:Node:NodeName"] = "central-a"` (central fixtures) / `"node-a"` (site fixtures) to every in-memory Host-boot config dict — grep the test project for `ScadaBridge:Node:` to find them all; the production `appsettings.{Central,Site}.json` and all docker per-node configs already set NodeName (verified), so no config-file changes are needed.
|
||||
5. Run, expect PASS:
|
||||
```bash
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~NodeOptionsValidator|FullyQualifiedName~CompositionRoot|FullyQualifiedName~MetricsEndpoint"
|
||||
dotnet build ZB.MOM.WW.ScadaBridge.slnx
|
||||
```
|
||||
6. Commit:
|
||||
```bash
|
||||
git add src/ZB.MOM.WW.ScadaBridge.Host tests/ZB.MOM.WW.ScadaBridge.Host.Tests
|
||||
git commit -m "fix(options): validate Host Node/Database/Logging options at startup — empty NodeName fails fast instead of NULLing SourceNode (plan R2-08 T7, arch-review 08r2 NF4)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 8: OperationTrackingOptions binding + the missing site `IOperationTrackingStore` registration (verify-then-fix)
|
||||
|
||||
**Classification:** standard — carries a **plan-authoring discovery** that may be a live functional gap, so investigate before coding
|
||||
**Estimated implement time:** ~5 min (after the ~5 min verification below confirms the gap)
|
||||
**Parallelizable with:** 1, 2, 3, 4, 5, 6, 10, 11, 12, 13 (blocked by 7 — same `Host/SiteServiceRegistration.cs`)
|
||||
**Files:**
|
||||
- Create: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingOptionsValidator.cs`
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs` (`AddSiteRuntime(connectionString)`, lines 36-82 — add the store registration)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs` (site-options block ~:134-152 — add the binding)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Tracking/OperationTrackingOptionsValidatorTests.cs`; extend `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CompositionRootTests.cs` (site-role fixture)
|
||||
|
||||
**Discovery (verified at plan time, main @ 8c888f13):** NF4 notes `OperationTrackingOptions` has no binding site. Verifying that surfaced something bigger: **no code anywhere in `src/` registers `IOperationTrackingStore` in DI** — yet `Host/Actors/AkkaHostedService.cs:1044` says "The store is registered by AddSiteRuntime on site composition roots" and `AuditLog/ServiceCollectionExtensions.cs:158-159` says "the operational tracking store is SITE-ONLY (registered by ZB.MOM.WW.ScadaBridge.SiteRuntime)". `AddSiteRuntime` (SiteRuntime SCE :36-82) registers no such thing. Every consumer resolves it with `GetService` (null-tolerant): `AkkaHostedService.cs:1048` (cached-drain scheduler never armed when null), `:1097-1099` (`PullSiteCalls` reconciliation seam never wired), `ScriptExecutionActor.cs:198` (`Tracking.Status` degraded), `AuditLog SCE :166` (forwarder degrades to audit-only). If the registration is truly absent, site-local cached-call tracking has been running in its degraded mode.
|
||||
|
||||
1. **Verify first** (do not trust this plan's grep over live behavior): `grep -rn "IOperationTrackingStore" src --include="*.cs"` and confirm no `AddSingleton`/descriptor registration exists. If one exists somewhere this plan missed — this task shrinks to the options binding + validator only; update the task notes accordingly.
|
||||
2. Write the failing composition-root lock — in the Host.Tests **site-role** fixture:
|
||||
```csharp
|
||||
[Fact]
|
||||
public void Site_Resolves_IOperationTrackingStore()
|
||||
{
|
||||
// Site Call Audit's site-local source of truth. AkkaHostedService and
|
||||
// ScriptRuntimeContext resolve this with GetService and silently degrade
|
||||
// when absent (audit-only telemetry, no cached-drain, no PullSiteCalls) —
|
||||
// so its presence on the SITE composition root must be locked here.
|
||||
var store = _host.Services.GetService<IOperationTrackingStore>();
|
||||
Assert.NotNull(store);
|
||||
}
|
||||
```
|
||||
Plus validator rejection tests: `ConnectionString = ""`, `RetentionDays = 0` (SiteRuntime.Tests; SiteRuntime already references `ZB.MOM.WW.Configuration` — `SiteRuntimeOptionsValidator` lives there).
|
||||
3. Run, expect FAIL:
|
||||
```bash
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~Site_Resolves_IOperationTrackingStore"
|
||||
```
|
||||
4. Implement:
|
||||
- `OperationTrackingOptionsValidator` (`ScadaBridge:OperationTracking` key prefix; `ConnectionString` non-empty, `RetentionDays > 0`).
|
||||
- In `AddSiteRuntime(connectionString)`: `services.AddSingleton<IOperationTrackingStore, OperationTrackingStore>();` (ctor takes `IOptions<OperationTrackingOptions>` + `ILogger` — verified at `OperationTrackingStore.cs:58-60`). Note in a comment that this makes the two existing "registered by AddSiteRuntime" claims true again.
|
||||
- In `Host/SiteServiceRegistration.cs` site-options block (~:134-152): `AddOptions<OperationTrackingOptions>().Bind(config.GetSection("ScadaBridge:OperationTracking")).ValidateOnStart()` + `TryAddEnumerable` the validator — same shape as its `SiteRuntimeOptions` neighbor.
|
||||
- Check `OperationTrackingStore`'s construction path: if the ctor eagerly opens/creates the SQLite file, point the Host.Tests site fixture at a temp path (`["ScadaBridge:OperationTracking:ConnectionString"] = $"Data Source={Path.Combine(Path.GetTempPath(), ...)}"`) so test runs don't litter `site-tracking.db` in the test cwd.
|
||||
5. Run, expect PASS — and because this un-degrades real behavior, run the consumer suites too:
|
||||
```bash
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~CompositionRoot"
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~Tracking"
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter "FullyQualifiedName~Telemetry"
|
||||
```
|
||||
6. If the gap was real (step 1 confirmed), flag it in the commit body — it is a functional fix riding a hygiene plan, and the user should know:
|
||||
```bash
|
||||
git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests tests/ZB.MOM.WW.ScadaBridge.Host.Tests
|
||||
git commit -m "fix(options): bind + validate OperationTrackingOptions; register the site IOperationTrackingStore that two comments claimed AddSiteRuntime provides (plan R2-08 T8, arch-review 08r2 NF4)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 9: NF8 — canonicalize the Communication/DataConnection section names, drop the duplicate Host bindings
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1, 2, 3, 4, 5, 6, 10, 11, 12, 13 (blocked by 8 — same `Host/SiteServiceRegistration.cs`)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Communication/ServiceCollectionExtensions.cs` (line 16), `src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptionsValidator.cs` (message prefixes)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/ServiceCollectionExtensions.cs` (lines 18, 27-31), `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/DataConnectionOptionsValidator.cs` (message prefixes, if any name the section)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs` (delete lines 140, 166, 171 — re-check offsets after Tasks 7/8)
|
||||
- Test: extend `tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/CommunicationOptionsValidatorTests.cs` (or a new `CommunicationOptionsBindingTests.cs`) and the DCL equivalent
|
||||
|
||||
**Context (verified — the drift NF8 warned about has already happened):** every real appsettings file nests these sections under `ScadaBridge:` (`Host/appsettings.Site.json:2` root, `Communication` at :37, `DataConnection` at :28; same in all docker per-node configs). But `AddCommunication()` binds `BindConfiguration("Communication")` (Communication SCE :16) and `AddDataConnectionLayer()` binds `"DataConnectionLayer"` (DCL SCE :18) — **sections that exist nowhere**. The options only get real values because the Host's duplicate `Configure<>` calls (`SiteServiceRegistration.cs:140/:166`) bind the true `ScadaBridge:*` sections. So "delete the Host copy" alone (the report's shorthand) would silently revert both components to defaults — the fix is: canonicalize the SCE section names to the real ones, THEN delete the Host duplicates. `NotificationOptions` is the clean case: its SCE already binds `"ScadaBridge:Notification"` (NotificationService SCE :23), so the Host duplicate at :171 is purely redundant — but `AddNotificationService()` is central-only (site path comment at `SiteServiceRegistration.cs:37-41`), so before deleting :171, grep for any site-path consumer of `IOptions<NotificationOptions>`; if none exists (expected — sites don't deliver), delete it.
|
||||
|
||||
1. Write the failing binding test (Communication.Tests):
|
||||
```csharp
|
||||
[Fact]
|
||||
public void AddCommunication_BindsTheScadaBridgeCommunicationSection()
|
||||
{
|
||||
// The real appsettings nest this section under "ScadaBridge:" — binding a
|
||||
// root-level "Communication" section reads config that no deployment writes
|
||||
// (arch-review 08 round 2 NF8: the Host duplicate was silently supplying the
|
||||
// real values). AddCommunication alone must bind the canonical section.
|
||||
var config = new ConfigurationBuilder().AddInMemoryCollection(new Dictionary<string, string?>
|
||||
{
|
||||
["ScadaBridge:Communication:DeploymentTimeout"] = "00:01:23",
|
||||
}).Build();
|
||||
var services = new ServiceCollection();
|
||||
services.AddSingleton<IConfiguration>(config);
|
||||
services.AddCommunication();
|
||||
using var sp = services.BuildServiceProvider();
|
||||
Assert.Equal(TimeSpan.FromSeconds(83), sp.GetRequiredService<IOptions<CommunicationOptions>>().Value.DeploymentTimeout);
|
||||
}
|
||||
```
|
||||
Mirror for DCL (`ScadaBridge:DataConnection:ReconnectInterval`). Run, expect FAIL (the SCE binds the wrong section, so the value stays at default).
|
||||
2. Implement:
|
||||
- Communication SCE :16 → `.BindConfiguration("ScadaBridge:Communication")`; update every `Communication:<Field>` message prefix in `CommunicationOptionsValidator` to `ScadaBridge:Communication:<Field>` (the messages exist to name the real config key).
|
||||
- DCL SCE :18 → `.BindConfiguration("ScadaBridge:DataConnection")`; same prefix sweep in its validator. For `OpcUaGlobalOptions`/`MxGatewayGlobalOptions` (:27-31, bare `"OpcUa"`/`"MxGateway"`): grep repo-wide (appsettings + tests + docs) for those bare sections first; no hit is expected (verified: none in any appsettings) — then canonicalize to `"ScadaBridge:OpcUa"`/`"ScadaBridge:MxGateway"` and note the sections in the two options classes' XML docs.
|
||||
- Delete the three Host duplicates (`SiteServiceRegistration.cs:140` DataConnection, `:166` Communication, `:171` Notification — after the consumer grep above).
|
||||
3. Run, expect PASS — plus both composition roots (this is exactly the change that could silently zero a timeout if the section name is wrong):
|
||||
```bash
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.Communication.Tests --filter "FullyQualifiedName~Options"
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~Options"
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~CompositionRoot"
|
||||
dotnet build ZB.MOM.WW.ScadaBridge.slnx
|
||||
```
|
||||
4. Live-config sanity (cheap, catches a typo'd section name better than any unit test): `bash docker/deploy.sh` is NOT required for this plan, but at minimum re-read one docker site config and one central config and confirm every key under `ScadaBridge:Communication` / `ScadaBridge:DataConnection` corresponds to a `CommunicationOptions`/`DataConnectionOptions` property.
|
||||
5. Commit:
|
||||
```bash
|
||||
git add src/ZB.MOM.WW.ScadaBridge.Communication src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs tests/ZB.MOM.WW.ScadaBridge.Communication.Tests tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests
|
||||
git commit -m "fix(options): canonicalize Communication/DataConnection config sections to ScadaBridge:*; drop the duplicate Host bindings that were masking the drift (plan R2-08 T9, arch-review 08r2 NF8)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 10: Re-consolidate deferral tracking into the canonical register
|
||||
|
||||
**Classification:** trivial (documentation)
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1, 2, 3, 4, 5, 6, 7, 8, 9, 12, 13
|
||||
**Files:**
|
||||
- Modify: `docs/plans/2026-07-08-deferred-work-register.md` (Fix-now table lines 7-16; "New deferrals" table lines 45-51)
|
||||
- Modify: `archreview/plans/00-MASTER-TRACKER.md` (registry section, lines 137-147)
|
||||
|
||||
Four edits, all specified by NF5 + the register's own rules:
|
||||
|
||||
1. **Prune the landed Fix-now rows** (the register's stated rule: "removed from this table when that plan's task lands" — all 7 landed). Replace the Fix-now table body with a single line: `All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round 2, 2026-07-12) — rows removed per the rule above; see the round-2 report §1 for the per-item evidence.`
|
||||
2. **Add the two tracker-only live items as first-class register rows** (a new `## Deferred — operational risk (from the initiative tracker, folded in 2026-07-12)` section, same column shape as the Deferred table):
|
||||
- **SBR oldest-crash total-outage gap** — 2-node `keep-oldest` downs the partition without the oldest, so a hard crash of the ACTIVE (oldest) central node makes the standby self-down (~10s) → total central outage until restart; only a younger-node crash fails over. Where noted: `archreview/plans/00-MASTER-TRACKER.md:143` + auto-memory. Rationale: remedy is a production SBR topology/strategy decision (keep-majority + 3rd lighthouse node, static-quorum, or an accepted-risk note) — **owner: user decision**, not silently changeable. Revisit trigger: before the next production deployment that adds a central node, or the first real active-node crash.
|
||||
- **`deploy/wonder-app-vd03/` overlay edits unapplied** — `appsettings.Central.json` needs `AllowSingleNodeCluster: true` + phantom-seed removal + `NodeName: central-a`; `install.ps1` needs `sc.exe failure` recovery actions. The artifact directory is intentionally untracked (production config out of source control), so the repo cannot ship the fix. Where noted: `00-MASTER-TRACKER.md:147` (PLAN-01 T16/T20/T23). Rationale: needs on-host access; without `NodeName` that deployment's audit rows stamp NULL `SourceNode` (partially mitigated once Task 7 lands — the host will now **fail at boot** with a key-naming error instead of silently NULLing, so applying the overlay becomes mandatory at next upgrade). Owner: whoever maintains the host (user). Revisit trigger: next wonder-app-vd03 deployment/upgrade — **the Task 7 validator makes this row unskippable then**.
|
||||
3. **Fix the fired revisit trigger** — the "Failover-timing + broader perf envelope" row (line 51): its trigger "PLAN-01 rig landing" fired on 2026-07-08 (`tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/TwoNodeClusterFixture.cs`), before the placeholder even shipped, and nobody noticed (NF2). Rewrite the row: status → **trigger fired 2026-07-08**; owner → **PLAN-R2-01** (the round-2 cluster/host/failover plan owns wiring `FailoverTimingTests` to the fixture or documenting the true blocker — reference its failover-envelope measurement task by number once that plan file exists; if PLAN-R2-01 is not yet written when this task executes, cite it as `archreview/plans/PLAN-R2-01-*.md (failover-envelope measurement task)`).
|
||||
4. **Subordinate the tracker's registry** — in `00-MASTER-TRACKER.md`, immediately under the `## Deferred / Won't-Fix Registry (initiative-level)` heading (line 137), add: `> 2026-07-12 (plan R2-08 T10): the two live items below (SBR oldest-crash gap, wonder-app-vd03 overlay) now have canonical rows in docs/plans/2026-07-08-deferred-work-register.md — that register is the single tracking place; the subsections below remain as the historical narrative.` Do not delete the narrative subsections (they carry the full evidence).
|
||||
5. Verify: `grep -n "SBR\|wonder-app-vd03" docs/plans/2026-07-08-deferred-work-register.md` → both new rows present; `grep -c "PLAN-08 Task 1 |" docs/plans/2026-07-08-deferred-work-register.md` → 0 (Fix-now rows gone); `grep -n "PLAN-R2-01" docs/plans/2026-07-08-deferred-work-register.md` → the repointed trigger row; `grep -n "single tracking place" archreview/plans/00-MASTER-TRACKER.md` → the subordination note.
|
||||
6. Commit:
|
||||
```bash
|
||||
git add docs/plans/2026-07-08-deferred-work-register.md archreview/plans/00-MASTER-TRACKER.md
|
||||
git commit -m "docs(plans): re-consolidate deferral tracking — SBR gap + wonder-app overlay rows into the register, prune landed Fix-now rows, repoint the fired perf-envelope trigger at PLAN-R2-01 (plan R2-08 T10, arch-review 08r2 NF5)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 11: Delete the drifted root `deferred.md` snapshot — `needs-user`
|
||||
|
||||
**Classification:** trivial (hygiene); **`needs-user`** (deletes an untracked file — unrecoverable)
|
||||
**Estimated implement time:** ~2 min
|
||||
**Parallelizable with:** 1, 2, 3, 4, 5, 6, 7, 8, 9, 12, 13 (blocked by 10 — the register must absorb/verify everything first)
|
||||
**Files:**
|
||||
- Delete (untracked): `deferred.md` (repo root)
|
||||
|
||||
`deferred.md` is a 2026-07-10 snapshot of the register that has already drifted (it counts "12 open deferrals" vs the register's ~10, restructures the sections, and its two ✅-SHIPPED rows duplicate the register's Resolved table). Duplicated state is how the trigger in NF2 got missed — it goes.
|
||||
|
||||
1. Diff it against the register for the user's benefit and confirm **nothing in `deferred.md` is absent from the post-Task-10 register** (the only unique content at plan-authoring time was cosmetic regrouping + the shipped-row prose, both already in the register's Resolved section). If the executor finds a genuinely unique fact, fold it into the register first (amend Task 10's commit).
|
||||
2. Show the user the file (it is untracked — deletion is unrecoverable) and get confirmation.
|
||||
3. `rm /Users/dohertj2/Desktop/ScadaBridge/deferred.md`
|
||||
4. Verify: `ls deferred.md 2>&1` → no such file; `git status --short` no longer lists it. No commit (untracked file).
|
||||
|
||||
---
|
||||
|
||||
### Task 12: Fix the stale Transport area comment in `EntitySerializer.FromBundleContent`
|
||||
|
||||
**Classification:** trivial (comment drift)
|
||||
**Estimated implement time:** ~3 min
|
||||
**Parallelizable with:** 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Serialization/EntitySerializer.cs` (lines 553-554)
|
||||
|
||||
`:553-554` still says "Areas don't travel (see export TODO), so AreaId stays null" — but areas DO travel by name since PLAN-05 (export resolves `AreaId` → `AreaNameById` at `:265-273`; the importer resolves-or-creates), and no TODO exists anymore. The `AreaId = null` at `:572` is still intentionally correct for this helper — `FromBundleContent` is off the importer's path (BundleImporter walks the raw DTO) and has no target-DB area-name→id map, exactly like the placeholder `DataConnectionId = 0` the same comment block already explains for connection bindings.
|
||||
|
||||
1. Replace the stale sentence with the truth, riding the existing explanation:
|
||||
```csharp
|
||||
// Instances: resolve template + site by name/identifier. Areas DO travel —
|
||||
// by NAME on InstanceDto.AreaName (export resolves AreaId→AreaNameById; the
|
||||
// importer resolves-or-creates under the target site). This helper still
|
||||
// materializes AreaId = null for the same reason the connection-binding FK
|
||||
// below is a placeholder: FromBundleContent is NOT on the importer's path
|
||||
// (BundleImporter walks the raw DTO) and has no target-DB name→id map to
|
||||
// resolve against. Round-trip fidelity for the area lives on the DTO, not
|
||||
// on this materialized entity. Connection bindings keep their ...
|
||||
```
|
||||
(Splice so the existing binding explanation at :555-562 continues unduplicated.)
|
||||
2. Verify (comment task — verification replaces a test):
|
||||
```bash
|
||||
grep -n "export TODO" src/ZB.MOM.WW.ScadaBridge.Transport/Serialization/EntitySerializer.cs # expect: no hits
|
||||
grep -c "Areas DO travel" src/ZB.MOM.WW.ScadaBridge.Transport/Serialization/EntitySerializer.cs # expect: 1
|
||||
dotnet build src/ZB.MOM.WW.ScadaBridge.Transport
|
||||
```
|
||||
3. Commit:
|
||||
```bash
|
||||
git add src/ZB.MOM.WW.ScadaBridge.Transport/Serialization/EntitySerializer.cs
|
||||
git commit -m "chore(hygiene): fix stale area-export comment in EntitySerializer.FromBundleContent — areas travel by name since PLAN-05 (plan R2-08 T12, arch-review 08r2 NF6)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 13: Commit the orphaned MES plan doc; remove the generated root reports — `needs-user` (deletions only)
|
||||
|
||||
**Classification:** trivial (hygiene); the two report deletions are **`needs-user`** (untracked — unrecoverable)
|
||||
**Estimated implement time:** ~3 min
|
||||
**Parallelizable with:** 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12
|
||||
**Files:**
|
||||
- Commit (currently untracked): `docs/plans/2026-06-30-mes-alarm-status-api.md`
|
||||
- Delete (untracked, user-confirmed): `ScadaBridge-docs-fixed.md`, `ScadaBridge-docs-issues.md` (root)
|
||||
- (`.claire/` is handled by Task 3's `.gitignore` entry — do not delete the directory; it holds tool worktrees)
|
||||
|
||||
1. The MES plan doc already carries the right header (`**Status:** Draft plan — NOT yet executed`), so it needs no edits — the repo's docs-as-spec convention says plan docs are committed decision records, and untracked it is invisible to every other session and one `git clean` from gone:
|
||||
```bash
|
||||
git add docs/plans/2026-06-30-mes-alarm-status-api.md
|
||||
git commit -m "docs(plans): commit the 2026-06-30 MES alarm-status API draft plan — untracked decision record (plan R2-08 T13, arch-review 08r2 NF7)"
|
||||
```
|
||||
2. `ScadaBridge-docs-fixed.md` (4 KB) / `ScadaBridge-docs-issues.md` (166 KB) are generated CommentChecker-style reports from 2026-07-10, already consumed. Show the user their first ~20 lines each, state they are untracked/unrecoverable, and on confirmation:
|
||||
```bash
|
||||
rm /Users/dohertj2/Desktop/ScadaBridge/ScadaBridge-docs-fixed.md /Users/dohertj2/Desktop/ScadaBridge/ScadaBridge-docs-issues.md
|
||||
```
|
||||
If the user wants them kept, move them to the session scratchpad or `~/` — not the repo root.
|
||||
3. Verify: `git status --short` shows neither report nor the plan doc as untracked; `git log --oneline -1 -- docs/plans/2026-06-30-mes-alarm-status-api.md` shows the new commit.
|
||||
|
||||
---
|
||||
|
||||
## Dependencies on other plans
|
||||
|
||||
- **PLAN-R2-01 (cluster/host/failover round 2)** owns NF2 in full: wiring `tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/Failover/FailoverTimingTests.cs:26-28` (whose skip reason is now stale) to the `TwoNodeClusterFixture` rig — or updating the skip text + register row with the true blocker. This plan's only NF2 touchpoint is Task 10 step 3, which repoints the register row's fired trigger at that plan's measurement task. **No file overlap** with PLAN-R2-01 is expected except `00-MASTER-TRACKER.md` (Task 10 adds one blockquote line) — coordinate merge order if both plans run concurrently.
|
||||
- No other round-2 plan shares files with this one at plan-authoring time; Tasks 7-9 all edit `Host/SiteServiceRegistration.cs`, which round-1 plans also touched — they are merged, so only the intra-plan 7→8→9 serialization matters.
|
||||
|
||||
## Execution order
|
||||
|
||||
1. **Task 1 first** (P0; starts the user-dependent rotation clock), Task 2 after it.
|
||||
2. **Tasks 3, 4, 5, 7, 10, 12, 13 in parallel** (disjoint files).
|
||||
3. **Task 6 after 5; Tasks 8 then 9 after 7; Task 11 after 10.**
|
||||
4. Finish with `dotnet build ZB.MOM.WW.ScadaBridge.slnx && dotnet test ZB.MOM.WW.ScadaBridge.slnx` — expect all 30 test projects green (the known environmental exceptions: the 8 Playwright E2E that need a live docker cluster, and the pre-existing `SiteConnectionUpGauge_*` gRPC gauge flake).
|
||||
|
||||
## Findings Coverage
|
||||
|
||||
| Report finding (08-conventions-tests-underdeveloped.md, round 2) | Severity | Coverage |
|
||||
|---|---|---|
|
||||
| NF1 — live production Inbound API key in `test.txt` at the repo root (plus, found on triage: live AD/SQL credentials in the gitignored `ldap_login.txt`/`sql_login.txt`/`sqllogin.txt`) | High | Tasks 1 (rotate via the shipped dual-key flow, `needs-user`), 2 (delete the credential files, `needs-user`), 3 (`.gitignore` + pre-commit guard) |
|
||||
| NF2 — failover envelope unmeasured; `FailoverTimingTests` skip precondition already satisfied | Medium | **Owned by PLAN-R2-01** (rig wiring / measurement). This plan: Task 10 step 3 repairs the register row whose "PLAN-01 rig landing" trigger fired unnoticed, repointing it at PLAN-R2-01's measurement task |
|
||||
| NF3 — CHANGELOG.md abandoned and factually wrong (`Operator`/`Engineer` claim false since M7) | Medium | Task 4 |
|
||||
| NF4 — options-validation residue: 6 AuditLog sub-options, Host Node/Database/Logging, unbound `OperationTrackingOptions` | Low | Tasks 5, 6 (AuditLog site + central sub-options), 7 (Host options; empty `NodeName` fails fast), 8 (OperationTracking binding + the discovered missing site `IOperationTrackingStore` registration) |
|
||||
| NF5 — deferral tracking re-scattered into three places (tracker registry, drifted root `deferred.md`, stale Fix-now rows) | Low | Tasks 10 (fold SBR gap + wonder-app overlay into the register, prune Fix-now, fix fired trigger, subordinate the tracker), 11 (delete root `deferred.md`, `needs-user`) |
|
||||
| NF6 — stale `EntitySerializer.cs:553-554` comment contradicts the shipped area export | Low | Task 12 |
|
||||
| NF7 — untracked residue: MES draft plan doc, root generated reports, `.claire/` | Low | Task 13 (commit the plan doc, remove the reports `needs-user`); Task 3 gitignores `.claire/`; `test.txt` itself is NF1/Task 2 |
|
||||
| NF8 — duplicate `DataConnectionOptions`/`CommunicationOptions`/`NotificationOptions` bindings in the Host (and, verified at plan time: the owning SCEs bind section names that exist in NO appsettings — the Host duplicates were the live bindings) | Low | Task 9 (canonicalize SCE section names to `ScadaBridge:*`, then delete the Host duplicates, with a binding-test lock) |
|
||||
@@ -0,0 +1,84 @@
|
||||
{
|
||||
"planPath": "archreview/plans/PLAN-R2-08-conventions-hygiene.md",
|
||||
"tasks": [
|
||||
{
|
||||
"id": 1,
|
||||
"subject": "Task 1: Rotate the exposed wonder-app-vd03 Inbound API key (dual-key flow) — needs-user",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"subject": "Task 2: Delete test.txt and triage the root credential files — needs-user",
|
||||
"status": "pending",
|
||||
"blockedBy": [1]
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"subject": "Task 3: Root secret-capture guards — .gitignore patterns + pre-commit secret scan",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"subject": "Task 4: CHANGELOG.md — correct the false role claim and refresh to reality",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"subject": "Task 5: Options validation — AuditLog site sub-options (SiteWriter, SiteTelemetry, SiteRetention)",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"subject": "Task 6: Options validation — AuditLog central sub-options (PartitionMaintenance, Purge, Reconciliation)",
|
||||
"status": "pending",
|
||||
"blockedBy": [5]
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
"subject": "Task 7: Options validation — Host NodeOptions / DatabaseOptions / LoggingOptions (empty NodeName fails fast)",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": 8,
|
||||
"subject": "Task 8: OperationTrackingOptions binding + the missing site IOperationTrackingStore registration (verify-then-fix)",
|
||||
"status": "pending",
|
||||
"blockedBy": [7]
|
||||
},
|
||||
{
|
||||
"id": 9,
|
||||
"subject": "Task 9: NF8 — canonicalize the Communication/DataConnection section names, drop the duplicate Host bindings",
|
||||
"status": "pending",
|
||||
"blockedBy": [8]
|
||||
},
|
||||
{
|
||||
"id": 10,
|
||||
"subject": "Task 10: Re-consolidate deferral tracking into the canonical register",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": 11,
|
||||
"subject": "Task 11: Delete the drifted root deferred.md snapshot — needs-user",
|
||||
"status": "pending",
|
||||
"blockedBy": [10]
|
||||
},
|
||||
{
|
||||
"id": 12,
|
||||
"subject": "Task 12: Fix the stale Transport area comment in EntitySerializer.FromBundleContent",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": 13,
|
||||
"subject": "Task 13: Commit the orphaned MES plan doc; remove the generated root reports — needs-user (deletions only)",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
}
|
||||
],
|
||||
"lastUpdated": "2026-07-13T03:30:22Z"
|
||||
}
|
||||
Reference in New Issue
Block a user