510 lines
28 KiB
Markdown
510 lines
28 KiB
Markdown
# Perf Follow-ups Round 3 Implementation Plan
|
||
|
||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:subagent-driven-development
|
||
> to implement this plan task-by-task (per the standing preference: Opus implementers,
|
||
> classification-driven review chains).
|
||
|
||
**Goal:** Close the actionable items in the round-2 follow-up register
|
||
(`docs/plans/2026-08-17-perf-followups-r2.md` §"Follow-up register") and run the
|
||
still-unexercised local live gates (FetchAndCache legs, alarm-bearing surgical ratchet,
|
||
ACL-enabled enforcement, docker-dev first-POST-404 catch), on branch `perf/followups-r3`
|
||
off master merge `49d05bc2`.
|
||
|
||
**Architecture:** Two surgical code fixes in already-reviewed code (the ACL cache's
|
||
single-shot CAS becomes a monotonic-max loop; S7 read-pass health becomes worst-state-wins
|
||
aggregation), one doc-staleness fix found during exploration, and four live gates on the
|
||
local docker-dev rig (OrbStack) sequenced on one rig session. No new subsystems.
|
||
|
||
**Tech Stack:** .NET 10 / C#, xUnit + Shouldly, docker-dev rig (OrbStack), Traefik v3.1
|
||
access log, Client.CLI, Chrome browser automation for AdminUI authoring.
|
||
|
||
**Out of scope (recorded, deliberately not here):**
|
||
- **TM/CT decode at a non-zero device number** — needs a real S7 CPU; the test exists and
|
||
is env-gated on `S7_TIMER_NUMBER_ADDRESSING=1`. Nothing to do until hardware exists.
|
||
- **Cross-process plant-vs-reap race on the shared SQL server** — accepted in r2; the 24 h
|
||
age floor is the mitigation. No work.
|
||
- **Modbus per-tag health last-writer-wins** — Task 2 fixes S7 only (the recorded item).
|
||
Modbus has the same shape (`ModbusDriver.ReadAsync:493/511/518` + the coalesced path);
|
||
fixing it here doubles the blast radius of a watchdog-adjacent change. Recorded in this
|
||
round's register as a deliberate asymmetry.
|
||
- **S7 write-path per-item health** (`S7Driver.cs:1575/1633/1645/1654`) — same
|
||
last-writer-wins shape on `WriteAsync`; out of scope for the same reason, recorded.
|
||
- Historian end-to-end live gate — needs the `wonder-sql-vd03` VPN (user-driven).
|
||
- HistorianGateway batch `WriteLiveValues` RPC — cross-repo.
|
||
- Load-scale throughput — needs a load harness; not defined enough to gate.
|
||
|
||
**Execution rules (unchanged from prior rounds):** shared checkout — explicit-pathspec
|
||
`git add`/`git commit -- <files>` only (never a bare commit); never checkout/switch/reset/
|
||
stash/clean; retry on `index.lock` and CS2012/MSB3030 build races; reviewers judge commits
|
||
via `git show`; falsification mutations applied AND reverted via Edit (never git), run
|
||
against a freshly built binary (never `--no-build` after a mutation), and labeled honestly
|
||
("pin, not red-first" where the behaviour predates the test; "probabilistic red" where the
|
||
mutation trips a race). Live-gate rig work is serial — one rig session, Tasks 4→5→6→7 in
|
||
that order, images built from the branch tip AFTER Tasks 1–3 land so the gates also
|
||
regression the new code.
|
||
|
||
---
|
||
|
||
### Task 1: ACL `InvalidateIfStale` — monotonic-max CAS loop (publication never deferred)
|
||
|
||
**Classification:** high-risk
|
||
**Estimated implement time:** ~4 min
|
||
**Parallelizable with:** Tasks 2, 3
|
||
|
||
**Files:**
|
||
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/Security/SessionAclDecisionCache.cs:453-461`
|
||
(+ the XML remarks block `:430-452`)
|
||
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/SessionAclDecisionCacheTests.cs`
|
||
|
||
**Context.** The r2 register records: the single-shot CAS in `InvalidateIfStale` can defer
|
||
stamp publication by one call. Two callers read the same `seen` baseline; gen-N wins the
|
||
CAS, gen-N+1's CAS fails against the moved stamp and falls out with **no retry** — N+1 is
|
||
not published on that call, so its `TryGet` misses and its `Set` is dropped at `:223`.
|
||
Benign (verdict safety lives in the per-shard generation gate), but it costs a lost store
|
||
exactly in the post-deploy window. Current code:
|
||
|
||
```csharp
|
||
private void InvalidateIfStale(long generation)
|
||
{
|
||
var seen = Interlocked.Read(ref _generation);
|
||
if (generation <= seen) return;
|
||
if (Interlocked.CompareExchange(ref _generation, generation, seen) == seen) ClearAll();
|
||
}
|
||
```
|
||
|
||
**Step 1: Write the racing pin (red against current code — probabilistic red, say so).**
|
||
|
||
```csharp
|
||
[Fact]
|
||
public void Concurrent_lookups_at_mixed_generations_always_publish_the_highest()
|
||
{
|
||
// The single-shot CAS defect: when gen-1 wins the shared-baseline race, gen-2's CAS
|
||
// fails and is never retried, leaving Generation == 1. Red is probabilistic per
|
||
// round; 200 barrier-synchronised rounds make it effectively certain.
|
||
for (var round = 0; round < 200; round++)
|
||
{
|
||
var cache = /* fresh cache, generation seed -1 */;
|
||
using var barrier = new Barrier(2);
|
||
var low = Task.Run(() => { barrier.SignalAndWait(); /* TryGet at generation 1 */ });
|
||
var high = Task.Run(() => { barrier.SignalAndWait(); /* TryGet at generation 2 */ });
|
||
Task.WaitAll(low, high);
|
||
cache.Generation.ShouldBe(2L,
|
||
$"round {round}: the newer generation must publish even when it loses the CAS race");
|
||
}
|
||
}
|
||
```
|
||
|
||
Use the ordinary `TryGet` lookup path (as the straggler test at `:286` does), NOT
|
||
`AdvanceGenerationForTest` — the defect lives in the lookup path's invalidation.
|
||
|
||
**Step 2: Run it, observe red.** `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests --filter "FullyQualifiedName~Concurrent_lookups_at_mixed_generations"` — expect FAIL within a few rounds. If 200 rounds pass green against unmodified code, the test is not exercising the race — fix the test, do not proceed.
|
||
|
||
**Step 3: Implement the monotonic-max loop.**
|
||
|
||
```csharp
|
||
private void InvalidateIfStale(long generation)
|
||
{
|
||
while (true)
|
||
{
|
||
var seen = Interlocked.Read(ref _generation);
|
||
if (generation <= seen) return; // guard re-checked per attempt
|
||
if (Interlocked.CompareExchange(ref _generation, generation, seen) == seen)
|
||
{
|
||
ClearAll();
|
||
return;
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
Invariants (update the XML remarks `:430-452` to say all of this):
|
||
- The `<=` guard is INSIDE the loop against a fresh read — a straggler that lost a CAS
|
||
re-reads the newer stamp and exits; hoisting the guard reintroduces the 98d6a190 defect.
|
||
- Only the thread whose CAS moved the stamp clears (elected clear preserved). A rollover
|
||
now legitimately clears once per distinct generation (N then N+1) — bounded by
|
||
generations, not request volume.
|
||
- The loop self-terminates: each iteration exits via the guard or CASes against a strictly
|
||
larger `seen`, and `_generation` only rises.
|
||
- The `-1` seed still admits generation 0.
|
||
|
||
**Step 4: Run the new test + the full cache suite.** All 18 green (17 existing + 1 new),
|
||
especially `A_straggler_from_an_older_generation_neither_clears_nor_rolls_the_stamp_back`
|
||
(`:286`) and `No_hit_ever_returns_a_verdict_from_another_generation` (`:363`).
|
||
|
||
**Step 5: Falsify.** Via Edit, revert the loop to the single-shot form (fresh build, not
|
||
`--no-build`) → the new test must go red (probabilistic; report the failing round). Revert
|
||
via Edit. Separately, move the `<=` guard outside the loop → the straggler test at `:286`
|
||
must go red. Revert.
|
||
|
||
**Step 6: Commit.**
|
||
```bash
|
||
git add -- src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/Security/SessionAclDecisionCache.cs \
|
||
tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/SessionAclDecisionCacheTests.cs
|
||
git commit -m "fix(acl): make InvalidateIfStale a monotonic-max CAS loop — stamp publication is never deferred" -- <same paths>
|
||
```
|
||
|
||
---
|
||
|
||
### Task 2: S7 read-pass health — worst-state-wins aggregation
|
||
|
||
**Classification:** high-risk
|
||
**Estimated implement time:** ~5 min
|
||
**Parallelizable with:** Tasks 1, 3
|
||
|
||
**Files:**
|
||
- Modify: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.S7/S7Driver.cs` — per-item health sites
|
||
`:647` (per-tag loop), `:828` (multi-var decode loop), `:941` (span member loop),
|
||
`RecordReadFailure:1010-1046`, plus one end-of-pass write in `ReadAsync`
|
||
- Test: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.S7.Tests/S7SpanReadParityTests.cs`
|
||
(flip the `:423` pin; doc-comment `:389-393` rewritten; new tests)
|
||
|
||
**Context.** The r2 register records: per-item decode health is last-writer-wins — a mixed
|
||
chunk reports whatever the LAST item's outcome was, so `Healthy` can be reported for a
|
||
pass that had failures (pinned at `:396-423`), and the `:1023` access-denied `Faulted` can
|
||
be overwritten by a later good item. Fix = aggregate one outcome per `ReadAsync` pass.
|
||
|
||
**Watchdog contract (MUST hold — CLAUDE.md comms-loss section):** `Degraded` means
|
||
connected-but-some-bad; `LastSuccessfulRead` advances on ANY success and is preserved on
|
||
failure. The aggregate write must advance `LastSuccessfulRead` when ANY item succeeded,
|
||
even when the state lands `Degraded`/`Faulted` — gating it on `Healthy` turns the fix into
|
||
a false-positive reconnect storm.
|
||
|
||
**Step 1: Red tests first** (the flip IS red-first; the rest are new):
|
||
1. Flip `:423`: `One_undecodable_multi_var_item...` now expects `Degraded` (rename to say
|
||
worst-state-wins; rewrite doc-comment `:389-393` — the last-writer-wins pin is being
|
||
deliberately retired).
|
||
2. New: a mixed multi-var pass reports `Degraded` AND `LastSuccessfulRead` is advanced
|
||
(non-null / newer than the pre-pass value).
|
||
3. New: span path — one bad member decode then a good member ⇒ `Degraded` (today: Healthy).
|
||
4. New: an access-denied `PlcException` on one tag followed by a success ⇒ state stays
|
||
`Faulted` (ordering `Healthy < Degraded < Faulted`), `LastSuccessfulRead` advanced.
|
||
Use the per-tag path if the multi-var fake can't produce a per-item PlcException.
|
||
5. Must-stay-green: `:326` (`A_failing_multi_var_chunk_falls_back...`, asserts Healthy at
|
||
`:346`) and `:432` (short answer, Healthy at `:447`) — a multi-var REQUEST fault whose
|
||
tags are re-served Good by the per-tag tail is NOT a failed outcome. The chunk-fault
|
||
catch (`:806-814`) stays outside the aggregate.
|
||
|
||
**Step 2: Run, observe red on 1–4** (each for its own reason; report per-test).
|
||
|
||
**Step 3: Implement.** A small per-pass accumulator threaded through the pass:
|
||
|
||
```csharp
|
||
private sealed class ReadPassOutcome
|
||
{
|
||
public bool AnySuccess;
|
||
public DriverState WorstFailure = DriverState.Healthy; // Healthy = none recorded
|
||
public string? FirstError; // first failure is causal
|
||
public void RecordSuccess() => AnySuccess = true;
|
||
public void RecordFailure(DriverState state, string message)
|
||
{
|
||
if (Rank(state) > Rank(WorstFailure)) WorstFailure = state;
|
||
FirstError ??= message;
|
||
static int Rank(DriverState s) => s switch
|
||
{
|
||
DriverState.Faulted => 2, DriverState.Degraded => 1, _ => 0
|
||
};
|
||
}
|
||
}
|
||
```
|
||
|
||
- The three success sites (`:647`, `:828`, `:941`) call `outcome.RecordSuccess()` instead
|
||
of assigning `_health`.
|
||
- `RecordReadFailure` takes the outcome, records `(Degraded|Faulted, message)` into it
|
||
instead of assigning `_health`, and KEEPS its `_plcDead` / `MarkConnectionDeadIfFatal`
|
||
side effects immediate. It still returns the per-item status code.
|
||
- One `_health` write at the end of the `ReadAsync` try body (NOT `finally` — the OCE
|
||
arms at `:649`/`:801`/`:920` rethrow and must skip it):
|
||
|
||
```csharp
|
||
_health = new DriverHealth(
|
||
outcome.WorstFailure == DriverState.Healthy
|
||
? DriverState.Healthy : outcome.WorstFailure,
|
||
outcome.AnySuccess ? now : _health.LastSuccessfulRead,
|
||
outcome.WorstFailure == DriverState.Healthy ? null : outcome.FirstError);
|
||
```
|
||
|
||
- Untouched: the `:567` connect-failure early return; `HandlePollError` (`:1980`); the
|
||
entire write path; init/shutdown sites. `ReadSpanAsync` / `ReadLeftoversByMultiVarAsync`
|
||
gain an outcome parameter — signature changes, no logic movement.
|
||
|
||
**Step 4: Run the full S7 suite** (`dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.S7.Tests`)
|
||
— all green including the parity anchor `:202` (all-members-fail ⇒ Degraded both paths).
|
||
|
||
**Step 5: Falsify.** Via Edit: (a) make the end-of-pass write use the LAST recorded state
|
||
instead of the worst → tests 1/3/4 red; (b) gate `LastSuccessfulRead` advancement on the
|
||
state being Healthy → test 2 red. Fresh builds. Revert via Edit.
|
||
|
||
**Step 6: Commit** (explicit pathspec, message
|
||
`fix(s7): read-pass health is worst-state-wins — a mixed pass can no longer report Healthy`).
|
||
|
||
---
|
||
|
||
### Task 3: Fix stale `OTOPCUA_CONFIG_MODE` docs
|
||
|
||
**Classification:** trivial
|
||
**Estimated implement time:** ~2 min
|
||
**Parallelizable with:** Tasks 1, 2
|
||
|
||
**Files:**
|
||
- Modify: `docs/Configuration.md` (~line 253) — "Flip only the site nodes with
|
||
`OTOPCUA_CONFIG_MODE=FetchAndCache` at `docker compose up`" is stale
|
||
- Modify: `CLAUDE.md` — same sentence in the `ConfigSource`/`ConfigServe` section
|
||
- Check-and-fix if matched: `grep -rn "OTOPCUA_CONFIG_MODE" docs/ docker-dev/ CLAUDE.md`
|
||
|
||
**Spec.** Phase 4 made `FetchAndCache` mandatory on driver-only nodes
|
||
(`ConfigSourceOptionsValidator` fails host start on `Direct`), and the rig's four site
|
||
nodes are hardcoded `ConfigSource__Mode: "FetchAndCache"` in
|
||
`docker-dev/docker-compose.yml:572-579/691-696/787-792/858-863`. The env var appears
|
||
nowhere in the compose file; setting it is a no-op. Rewrite both sentences to state: the
|
||
site nodes run `FetchAndCache` unconditionally (mandatory for driver-only nodes since
|
||
Phase 4); central stays `Direct` via its admin role. No review (doc-only). Commit with
|
||
explicit pathspec, message `docs(config): OTOPCUA_CONFIG_MODE is gone — site nodes are hardcoded FetchAndCache (Phase 4)`.
|
||
|
||
---
|
||
|
||
### Task 4: Live gate — docker-dev first-POST-404 catch + no-body POST binding
|
||
|
||
**Classification:** live-gate (rig session step 1 — FRESH BOOT window)
|
||
**Estimated implement time:** ~10 min wall (mostly rig boot)
|
||
**Parallelizable with:** none (rig serial; runs after Tasks 1–3 land, images built from branch tip)
|
||
|
||
**Files:**
|
||
- Possibly modify: `docker-dev/deploy.sh` (the header NOTE about no-body binding),
|
||
`docker-dev/README.md` (§"First-POST 404" — mechanism note if caught)
|
||
- Evidence: write to the scratchpad; Task 8 folds it into the plan's Outcomes
|
||
|
||
**Procedure:**
|
||
1. Fresh rig: `docker compose -f docker-dev/docker-compose.yml down -v` then
|
||
`up -d --build` (build mandatory — the branch tip must be in the images).
|
||
2. IMMEDIATELY run `./docker-dev/deploy.sh` (no body). Its 2 s `/health/ready` poll rides
|
||
through Traefik during the boot window — every poll is access-logged with
|
||
`defaultmode=keep`, and an unmatched request shows an EMPTY `RouterName`.
|
||
3. After the deploy completes (or fails), capture
|
||
`docker compose -f docker-dev/docker-compose.yml logs traefik` and extract every
|
||
non-2xx access-log line from the boot window. **If any 404 appears:** name the
|
||
mechanism from `RouterName`/`ServiceName`/timing (the r2 elimination says: AdminUI
|
||
cannot 404 that path; backend-down is 503; so 404 = no router matched at that instant
|
||
— presumably the window before the dynamic-config file provider loads or before the
|
||
health-checked service has a server). Update `docker-dev/README.md` §"First-POST 404"
|
||
with the observed mechanism. **If it does not reproduce:** record that honestly —
|
||
"did not reproduce under the readiness gate; register item stays open" — do NOT
|
||
invent a mechanism.
|
||
4. No-body POST binding: with the rig ready, run `./docker-dev/deploy.sh` bare and record
|
||
the HTTP status + response body. Determine whether the minimal-API `DeployRequest?`
|
||
binds null (and what the endpoint then does) or 400s. Update the header NOTE in
|
||
`deploy.sh` from "unverified" to the observed behaviour, and `README.md` if wrong.
|
||
5. Leave the rig UP for Task 5.
|
||
|
||
Commit any `deploy.sh`/`README.md` edits (explicit pathspec). Report: the access-log
|
||
excerpt, the no-body verdict, and whether the 404 reproduced.
|
||
|
||
---
|
||
|
||
### Task 5: Live gate — FetchAndCache legs on the running rig
|
||
|
||
**Classification:** live-gate (rig session step 2)
|
||
**Estimated implement time:** ~8 min wall
|
||
**Parallelizable with:** none (rig serial, after Task 4)
|
||
|
||
**Procedure** (rig already up from Task 4; a deploy already ran):
|
||
1. **Fetch + verify:** `docker compose -f docker-dev/docker-compose.yml logs site-a-1 |
|
||
grep -E "Fetched \+ verified artifact"` — expect
|
||
`Fetched + verified artifact for {DeploymentId} from http://central-… ({Bytes} bytes).`
|
||
2. **Served (central side):** `logs central-1 central-2 | grep "Served artifact"` —
|
||
`Served artifact for {DeploymentId} ({Bytes} bytes, {Chunks} chunks)`.
|
||
3. **Applied-from-fetch + cached:** `logs site-a-1 | grep -E "applied fetched deployment|cached applied deployment"` — both lines must appear (the `fetched` variant, NOT the
|
||
Direct `applied deployment` form).
|
||
4. **Boot-from-cache:** `docker compose restart site-a-1`, wait for boot, then
|
||
`logs site-a-1 | grep "FetchAndCache boot"` — expect
|
||
`restored served state for deployment {Id} (rev {Rev}) from the LocalDb pointer.`
|
||
5. **Absence proof:** on site-a-1 logs, assert NO occurrence of `entering Stale`,
|
||
`ConfigDb back`, or the Direct-form `applied deployment {Id} (rev` (without
|
||
`fetched`). A FetchAndCache node must never reach Stale.
|
||
6. Evidence excerpts to the scratchpad for Task 8. Rig stays UP for Task 6.
|
||
|
||
No commit expected. If any leg fails, STOP and report — that is a real regression, not a
|
||
gate formality.
|
||
|
||
---
|
||
|
||
### Task 6: Live gate — alarm-bearing surgical ratchet
|
||
|
||
**Classification:** live-gate (rig session step 3)
|
||
**Estimated implement time:** ~12 min wall
|
||
**Parallelizable with:** none (rig serial, after Task 5)
|
||
|
||
**Context.** The surgical-deploys classifier routes an alarm-bearing raw-tag delta
|
||
(`Alarm != null` on EITHER side) to `Rebuild`; a whitelisted non-alarm attribute delta
|
||
(`Writable` among them) is surgical. Never observed live. The ONLY seam is the applier
|
||
summary line (`AddressSpaceApplier.cs:322-326`):
|
||
`applied plan (kind={Kind}, …, surgicalRawTags={SurgicalRaw}, …, rebuild={Rebuild})`.
|
||
|
||
**Procedure:**
|
||
1. Pick TWO alarm-free raw tags in the rig's seeded config (browse `/raw` via AdminUI at
|
||
`http://localhost:9200`, login disabled — or read the ConfigDb `Tag` rows). Call them
|
||
TagS (surgical lever) and TagA (alarm lever).
|
||
2. **Surgical control first:** toggle TagS's Access level (Read ↔ ReadWrite) — via the
|
||
RawTagModal in the browser, or a direct ConfigDb UPDATE of the tag row (the rig's DB
|
||
on `10.100.0.35,14330`; acceptable for a classifier gate, note which route was used).
|
||
Deploy via `./docker-dev/deploy.sh`. Expect on central-1:
|
||
`kind=AttributeOnly, … surgicalRawTags=1, … rebuild=False`.
|
||
3. **Alarm add ⇒ rebuild:** add `"alarm": {"alarmType": "OffNormalAlarm", "severity": 500}`
|
||
to TagA's `TagConfig` JSON (raw-JSON textarea — available because the seeded device is
|
||
a Galaxy driver, which has no typed editor — or the same direct-DB route). Deploy.
|
||
Expect `kind=Rebuild, … surgicalRawTags=0, … rebuild=True`.
|
||
4. **Alarm remove ⇒ rebuild:** remove the `alarm` object, deploy, expect `rebuild=True`
|
||
again (Alarm != null on the PREVIOUS side also disqualifies).
|
||
5. **Restore:** revert TagS's access level, deploy once more (surgical again — also
|
||
re-proves the ratchet is per-deploy, not latched).
|
||
6. Evidence: the four applier lines (grep `docker compose logs central-1 site-a-1 |
|
||
grep "applied plan (kind="` — note which nodes applied). Scratchpad for Task 8. Rig
|
||
stays UP for Task 7.
|
||
|
||
No commit expected. Failure of the expected routing = STOP and report.
|
||
|
||
---
|
||
|
||
### Task 7: Live gate — ACL-enabled enforcement on site-a (re-shaped cache)
|
||
|
||
**Classification:** live-gate (rig session step 4)
|
||
**Estimated implement time:** ~20 min wall
|
||
**Parallelizable with:** none (rig serial, after Task 6)
|
||
|
||
**Context.** `Security:Acl:Enabled` has never been exercised live since the r1/r2 cache
|
||
re-shape (per-key shards, generation stamp, Task 1's CAS loop). Site-a is the env-gated
|
||
demo pair: `OTOPCUA_ACL_ENABLED=true` reaches site-a-1/-2 only
|
||
(`docker-compose.yml:585/:702`); site-b stays off as the A/B control. Site-a OPC UA:
|
||
host ports 4842/4843; site-b: 4844/4845. Keys/grants reach driver-only site nodes ONLY
|
||
via the deployment artifact — every authoring step needs a deploy after it.
|
||
|
||
**Procedure:**
|
||
1. Restart site-a gated: `OTOPCUA_ACL_ENABLED=true docker compose -f docker-dev/docker-compose.yml up -d site-a-1 site-a-2`. Confirm the boot Warning on both:
|
||
`node ACL enforcement is ENABLED — EVERY data-plane operation … requires a matching NodeAcl grant`.
|
||
2. Mint a key: AdminUI `/apikeys` (Chrome automation; login disabled) — Name
|
||
`r3-acl-gate`, Principal `r3gate`. COPY the one-time `otua_…` credential immediately.
|
||
3. Author a grant: `/clusters/{site-a cluster id}/acls/new` — Principal `r3gate`, scope
|
||
picked via the AclScopePickerModal on a WRITABLE raw tag (or its device — inheritance
|
||
covers the tag), permissions: the Operator bundle (must include Browse/Read/
|
||
WriteOperate). Save; expect no "authorizes nobody" warning.
|
||
4. Deploy (`./docker-dev/deploy.sh`) so the key + grant ride the artifact to site-a.
|
||
Confirm on site-a-1: `data-plane authority generation {G} installed (… keys=…, grants=…)`.
|
||
5. **The load-bearing ALLOW:** Client.CLI write through the grant —
|
||
`dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- write -u opc.tcp://localhost:4842 -n "<granted raw tag NodeId>" -v <value> -U apikey -P otua_…`
|
||
Expect success (the session carries NO server-wide role; the grant alone authorizes —
|
||
a deny-only gate would pass against a build where the ACL is never consulted).
|
||
Note: the rig's Galaxy driver has no live gateway, so the DEVICE write may fail
|
||
downstream — the gate is that the request passes the ACL (not `BadUserAccessDenied`);
|
||
any downstream Bad status is fine and should be reported as such.
|
||
6. **Deny controls:** (a) same write WITHOUT credentials (anonymous) ⇒ denied; (b) a
|
||
read/browse on a node OUTSIDE the granted scope with the key ⇒ withheld (grep
|
||
site-a-1 for `AclGate: denied {Operation} on {NodeId} for principal r3gate`).
|
||
7. **Revocation reaches the artifact path:** disable the key on `/apikeys`, deploy,
|
||
confirm a NEW `data-plane authority generation {G+1} installed` on site-a-1, and the
|
||
same CLI write now fails auth (`ApiKeyAuth: denied — no enabled, unexpired key`).
|
||
8. **Teardown:** restore site-a ungated (`docker compose up -d site-a-1 site-a-2` without
|
||
the env), optionally delete the gate key/grant, and `docker compose down` the rig
|
||
unless Task 8 needs it. Evidence excerpts to the scratchpad.
|
||
|
||
No commit expected. If Chrome automation of the scope picker proves impractical, direct
|
||
ConfigDb inserts for `ApiKey`/`NodeAcl` rows are acceptable ONLY as a fallback — say so,
|
||
and keep the key's peppered HMAC consistent with the rig pepper
|
||
(`docker-dev-apikey-pepper-not-for-production`); minting via the UI is strongly preferred
|
||
because it exercises the real authoring path.
|
||
|
||
---
|
||
|
||
### Task 9: Roslyn evaluator racing test — remove the timing assumption
|
||
|
||
**Classification:** standard
|
||
**Estimated implement time:** ~5 min
|
||
**Parallelizable with:** Task 10 (disjoint files); independent of the rig tasks
|
||
|
||
**Origin:** ScadaBridge deferred-work register row 38, relayed cross-session 2026-08-18
|
||
(coordinated by the ScadaBridge session on Joe's behalf).
|
||
|
||
**Files:**
|
||
- Modify: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/RoslynVirtualTagEvaluatorTests.cs`
|
||
(`Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed`, ~:391)
|
||
|
||
**Spec.** The test is load-fragile: it fails under full-solution parallel test load but
|
||
passes 3/3 in isolation. Its design assumes at most ONE `ClearCompiledScripts` lands
|
||
inside a single evaluation (so the product's one-shot retry suffices); under contention
|
||
two can land and the retry legitimately exhausts. Test-design issue, NOT a product bug.
|
||
Fix by removing the timing assumption while keeping the original intent (no
|
||
disposed-script failure ever ESCAPES the retry): either bound the clear cadence relative
|
||
to observed evaluation latency, or assert on retry-exhaustion behaviour instead of zero
|
||
failures — implementer picks whichever preserves intent with less machinery, documents
|
||
the choice in the test doc-comment, and demonstrates the test survives a
|
||
full-parallel-load run (run the containing suite alongside another heavy suite, or
|
||
repeat-run the suite ≥5×; report what was actually run).
|
||
|
||
---
|
||
|
||
### Task 10: LocalDb conditional registration + baseline + deregister self-heal
|
||
|
||
**Classification:** high-risk
|
||
**Estimated implement time:** ~8 min
|
||
**Parallelizable with:** Task 9 (disjoint files)
|
||
|
||
**Origin:** ScadaBridge deferred-work register row 39, relayed cross-session 2026-08-18.
|
||
ScadaBridge closed the same class with conditional registration + boot-time self-heal
|
||
(`Host/SiteLocalDbSetup.cs`, arch-review WP1.3/WP3.3); port that pattern here.
|
||
|
||
**Files:**
|
||
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSetup.cs` (`OnReady`,
|
||
registration at :63-66)
|
||
- Read for the predicate: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbRegistration.cs`
|
||
(`LocalDb:SyncListenPort`, `LocalDb:Replication:*` keys)
|
||
- Modify: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDbSetupTests.cs`
|
||
(the registered-table-set + ordering pins — update DELIBERATELY, say so in the diff)
|
||
- Modify: `CLAUDE.md` LocalDb section — the two bullets recording unconditional
|
||
registration as deliberate ("DeregisterReplicated exists and this repo deliberately
|
||
does not call it" / "Consequently, baselineExistingRows is not used either") must be
|
||
rewritten to record the new conditional shape and why it changed.
|
||
|
||
**Spec.**
|
||
- "Replication configured" = `PeerAddress` OR `ApiKey` present (OR, because only the
|
||
dialling half sets PeerAddress while the passive half carries just the key + listen
|
||
port — verify the exact key names against `LocalDbRegistration`).
|
||
- Configured ⇒ `RegisterReplicated(table, baselineExistingRows: true)` per table, in the
|
||
existing DDL→register order (the ordering is load-bearing and pinned) — seeds
|
||
row-versions at the LWW floor and flags snapshot resync, so late-enabled replication
|
||
on an established node converges on EXISTING rows, not just subsequent writes.
|
||
- NOT configured ⇒ `DeregisterReplicated` on every table at boot — idempotent self-heal:
|
||
drops stale triggers, prunes oplog/row-version rows, logs once at Information when it
|
||
actually cleaned something (and stays silent when there was nothing to clean).
|
||
- Registration stays per-table in a loop (0.3.0 made the snapshot flag per-table; the
|
||
loop preserves that granularity if the pin ever moves).
|
||
- **Pin note:** this repo is on `ZB.MOM.WW.LocalDb 0.2.1`, which has both
|
||
`baselineExistingRows` and `DeregisterReplicated`. Do NOT bump the pin unless a needed
|
||
API is genuinely absent at 0.2.1 — if it is, stop and report before bumping.
|
||
- **Traps (both hit by ScadaBridge):** the registered-table set is pinned by
|
||
`LocalDbSetupTests` — update the pins deliberately with the behaviour change named in
|
||
the test doc-comment; and deregistration must be pair-symmetric — the sync handshake
|
||
compares registered-table digests fail-closed, so flipping registration is a
|
||
both-nodes-together change. Record the pair-symmetry consequence in CLAUDE.md and in
|
||
the rollout note (docker-dev site-a has replication ON and was enabled from fresh, so
|
||
current impact is low; the change is correctness-before-someone-enables-late).
|
||
- Tests: pin the new predicate both ways (configured ⇒ registered with baseline;
|
||
unconfigured ⇒ deregistered/self-healed) and keep the DDL→register ordering pin green.
|
||
|
||
---
|
||
|
||
### Task 8: Close-out — sweep + Outcomes + register annotations
|
||
|
||
**Classification:** close-out (blockedBy: all)
|
||
**Estimated implement time:** ~15 min
|
||
|
||
1. Full build (`dotnet build ZB.MOM.WW.OtOpcUa.slnx`) — 0 errors; warning count vs r2's
|
||
904 explained.
|
||
2. Full unit sweep, every `*.Tests` project, per-project; reconcile the passed-count
|
||
delta against Tasks 1–2's added/changed tests exactly. DB suites must RUN (AdminUI
|
||
`0 skipped`, Configuration `0 skipped`).
|
||
3. Write `## Outcomes` on this plan: sweep record, per-task table, all four live-gate
|
||
evidence excerpts, falsification notes, and this round's register (Modbus + S7-write
|
||
asymmetry, TM/CT real-CPU, cross-process reap race carried; whatever Tasks 4–7 leave
|
||
open).
|
||
4. Annotate the r2 register in `2026-08-17-perf-followups-r2.md` append-only with
|
||
"Closed by <sha> (r3)" / "Gated live (r3)" sub-bullets.
|
||
5. Update the `.tasks.json` ledger; commit docs with explicit pathspec.
|