Joseph Doherty
895 lines
56 KiB
Markdown
895 lines
56 KiB
Markdown
# Perf Follow-ups Round 3 Implementation Plan
|
||
|
||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:subagent-driven-development
|
||
> to implement this plan task-by-task (per the standing preference: Opus implementers,
|
||
> classification-driven review chains).
|
||
|
||
**Goal:** Close the actionable items in the round-2 follow-up register
|
||
(`docs/plans/2026-08-17-perf-followups-r2.md` §"Follow-up register") and run the
|
||
still-unexercised local live gates (FetchAndCache legs, alarm-bearing surgical ratchet,
|
||
ACL-enabled enforcement, docker-dev first-POST-404 catch), on branch `perf/followups-r3`
|
||
off master merge `49d05bc2`.
|
||
|
||
**Architecture:** Two surgical code fixes in already-reviewed code (the ACL cache's
|
||
single-shot CAS becomes a monotonic-max loop; S7 read-pass health becomes worst-state-wins
|
||
aggregation), one doc-staleness fix found during exploration, and four live gates on the
|
||
local docker-dev rig (OrbStack) sequenced on one rig session. No new subsystems.
|
||
|
||
**Tech Stack:** .NET 10 / C#, xUnit + Shouldly, docker-dev rig (OrbStack), Traefik v3.1
|
||
access log, Client.CLI, Chrome browser automation for AdminUI authoring.
|
||
|
||
**Out of scope (recorded, deliberately not here):**
|
||
- **TM/CT decode at a non-zero device number** — needs a real S7 CPU; the test exists and
|
||
is env-gated on `S7_TIMER_NUMBER_ADDRESSING=1`. Nothing to do until hardware exists.
|
||
- **Cross-process plant-vs-reap race on the shared SQL server** — accepted in r2; the 24 h
|
||
age floor is the mitigation. No work.
|
||
- **Modbus per-tag health last-writer-wins** — Task 2 fixes S7 only (the recorded item).
|
||
Modbus has the same shape (`ModbusDriver.ReadAsync:493/511/518` + the coalesced path);
|
||
fixing it here doubles the blast radius of a watchdog-adjacent change. Recorded in this
|
||
round's register as a deliberate asymmetry.
|
||
- **S7 write-path per-item health** (`S7Driver.cs:1575/1633/1645/1654`) — same
|
||
last-writer-wins shape on `WriteAsync`; out of scope for the same reason, recorded.
|
||
- Historian end-to-end live gate — needs the `wonder-sql-vd03` VPN (user-driven).
|
||
- HistorianGateway batch `WriteLiveValues` RPC — cross-repo.
|
||
- Load-scale throughput — needs a load harness; not defined enough to gate.
|
||
|
||
**Execution rules (unchanged from prior rounds):** shared checkout — explicit-pathspec
|
||
`git add`/`git commit -- <files>` only (never a bare commit); never checkout/switch/reset/
|
||
stash/clean; retry on `index.lock` and CS2012/MSB3030 build races; reviewers judge commits
|
||
via `git show`; falsification mutations applied AND reverted via Edit (never git), run
|
||
against a freshly built binary (never `--no-build` after a mutation), and labeled honestly
|
||
("pin, not red-first" where the behaviour predates the test; "probabilistic red" where the
|
||
mutation trips a race). Live-gate rig work is serial — one rig session, Tasks 4→5→6→7 in
|
||
that order, images built from the branch tip AFTER Tasks 1–3 land so the gates also
|
||
regression the new code.
|
||
|
||
---
|
||
|
||
### Task 1: ACL `InvalidateIfStale` — monotonic-max CAS loop (publication never deferred)
|
||
|
||
**Classification:** high-risk
|
||
**Estimated implement time:** ~4 min
|
||
**Parallelizable with:** Tasks 2, 3
|
||
|
||
**Files:**
|
||
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/Security/SessionAclDecisionCache.cs:453-461`
|
||
(+ the XML remarks block `:430-452`)
|
||
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/SessionAclDecisionCacheTests.cs`
|
||
|
||
**Context.** The r2 register records: the single-shot CAS in `InvalidateIfStale` can defer
|
||
stamp publication by one call. Two callers read the same `seen` baseline; gen-N wins the
|
||
CAS, gen-N+1's CAS fails against the moved stamp and falls out with **no retry** — N+1 is
|
||
not published on that call, so its `TryGet` misses and its `Set` is dropped at `:223`.
|
||
Benign (verdict safety lives in the per-shard generation gate), but it costs a lost store
|
||
exactly in the post-deploy window. Current code:
|
||
|
||
```csharp
|
||
private void InvalidateIfStale(long generation)
|
||
{
|
||
var seen = Interlocked.Read(ref _generation);
|
||
if (generation <= seen) return;
|
||
if (Interlocked.CompareExchange(ref _generation, generation, seen) == seen) ClearAll();
|
||
}
|
||
```
|
||
|
||
**Step 1: Write the racing pin (red against current code — probabilistic red, say so).**
|
||
|
||
```csharp
|
||
[Fact]
|
||
public void Concurrent_lookups_at_mixed_generations_always_publish_the_highest()
|
||
{
|
||
// The single-shot CAS defect: when gen-1 wins the shared-baseline race, gen-2's CAS
|
||
// fails and is never retried, leaving Generation == 1. Red is probabilistic per
|
||
// round; 200 barrier-synchronised rounds make it effectively certain.
|
||
for (var round = 0; round < 200; round++)
|
||
{
|
||
var cache = /* fresh cache, generation seed -1 */;
|
||
using var barrier = new Barrier(2);
|
||
var low = Task.Run(() => { barrier.SignalAndWait(); /* TryGet at generation 1 */ });
|
||
var high = Task.Run(() => { barrier.SignalAndWait(); /* TryGet at generation 2 */ });
|
||
Task.WaitAll(low, high);
|
||
cache.Generation.ShouldBe(2L,
|
||
$"round {round}: the newer generation must publish even when it loses the CAS race");
|
||
}
|
||
}
|
||
```
|
||
|
||
Use the ordinary `TryGet` lookup path (as the straggler test at `:286` does), NOT
|
||
`AdvanceGenerationForTest` — the defect lives in the lookup path's invalidation.
|
||
|
||
**Step 2: Run it, observe red.** `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests --filter "FullyQualifiedName~Concurrent_lookups_at_mixed_generations"` — expect FAIL within a few rounds. If 200 rounds pass green against unmodified code, the test is not exercising the race — fix the test, do not proceed.
|
||
|
||
**Step 3: Implement the monotonic-max loop.**
|
||
|
||
```csharp
|
||
private void InvalidateIfStale(long generation)
|
||
{
|
||
while (true)
|
||
{
|
||
var seen = Interlocked.Read(ref _generation);
|
||
if (generation <= seen) return; // guard re-checked per attempt
|
||
if (Interlocked.CompareExchange(ref _generation, generation, seen) == seen)
|
||
{
|
||
ClearAll();
|
||
return;
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
Invariants (update the XML remarks `:430-452` to say all of this):
|
||
- The `<=` guard is INSIDE the loop against a fresh read — a straggler that lost a CAS
|
||
re-reads the newer stamp and exits; hoisting the guard reintroduces the 98d6a190 defect.
|
||
- Only the thread whose CAS moved the stamp clears (elected clear preserved). A rollover
|
||
now legitimately clears once per distinct generation (N then N+1) — bounded by
|
||
generations, not request volume.
|
||
- The loop self-terminates: each iteration exits via the guard or CASes against a strictly
|
||
larger `seen`, and `_generation` only rises.
|
||
- The `-1` seed still admits generation 0.
|
||
|
||
**Step 4: Run the new test + the full cache suite.** All 18 green (17 existing + 1 new),
|
||
especially `A_straggler_from_an_older_generation_neither_clears_nor_rolls_the_stamp_back`
|
||
(`:286`) and `No_hit_ever_returns_a_verdict_from_another_generation` (`:363`).
|
||
|
||
**Step 5: Falsify.** Via Edit, revert the loop to the single-shot form (fresh build, not
|
||
`--no-build`) → the new test must go red (probabilistic; report the failing round). Revert
|
||
via Edit. Separately, move the `<=` guard outside the loop → the straggler test at `:286`
|
||
must go red. Revert.
|
||
|
||
**Step 6: Commit.**
|
||
```bash
|
||
git add -- src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/Security/SessionAclDecisionCache.cs \
|
||
tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/SessionAclDecisionCacheTests.cs
|
||
git commit -m "fix(acl): make InvalidateIfStale a monotonic-max CAS loop — stamp publication is never deferred" -- <same paths>
|
||
```
|
||
|
||
---
|
||
|
||
### Task 2: S7 read-pass health — worst-state-wins aggregation
|
||
|
||
**Classification:** high-risk
|
||
**Estimated implement time:** ~5 min
|
||
**Parallelizable with:** Tasks 1, 3
|
||
|
||
**Files:**
|
||
- Modify: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.S7/S7Driver.cs` — per-item health sites
|
||
`:647` (per-tag loop), `:828` (multi-var decode loop), `:941` (span member loop),
|
||
`RecordReadFailure:1010-1046`, plus one end-of-pass write in `ReadAsync`
|
||
- Test: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.S7.Tests/S7SpanReadParityTests.cs`
|
||
(flip the `:423` pin; doc-comment `:389-393` rewritten; new tests)
|
||
|
||
**Context.** The r2 register records: per-item decode health is last-writer-wins — a mixed
|
||
chunk reports whatever the LAST item's outcome was, so `Healthy` can be reported for a
|
||
pass that had failures (pinned at `:396-423`), and the `:1023` access-denied `Faulted` can
|
||
be overwritten by a later good item. Fix = aggregate one outcome per `ReadAsync` pass.
|
||
|
||
**Watchdog contract (MUST hold — CLAUDE.md comms-loss section):** `Degraded` means
|
||
connected-but-some-bad; `LastSuccessfulRead` advances on ANY success and is preserved on
|
||
failure. The aggregate write must advance `LastSuccessfulRead` when ANY item succeeded,
|
||
even when the state lands `Degraded`/`Faulted` — gating it on `Healthy` turns the fix into
|
||
a false-positive reconnect storm.
|
||
|
||
**Step 1: Red tests first** (the flip IS red-first; the rest are new):
|
||
1. Flip `:423`: `One_undecodable_multi_var_item...` now expects `Degraded` (rename to say
|
||
worst-state-wins; rewrite doc-comment `:389-393` — the last-writer-wins pin is being
|
||
deliberately retired).
|
||
2. New: a mixed multi-var pass reports `Degraded` AND `LastSuccessfulRead` is advanced
|
||
(non-null / newer than the pre-pass value).
|
||
3. New: span path — one bad member decode then a good member ⇒ `Degraded` (today: Healthy).
|
||
4. New: an access-denied `PlcException` on one tag followed by a success ⇒ state stays
|
||
`Faulted` (ordering `Healthy < Degraded < Faulted`), `LastSuccessfulRead` advanced.
|
||
Use the per-tag path if the multi-var fake can't produce a per-item PlcException.
|
||
5. Must-stay-green: `:326` (`A_failing_multi_var_chunk_falls_back...`, asserts Healthy at
|
||
`:346`) and `:432` (short answer, Healthy at `:447`) — a multi-var REQUEST fault whose
|
||
tags are re-served Good by the per-tag tail is NOT a failed outcome. The chunk-fault
|
||
catch (`:806-814`) stays outside the aggregate.
|
||
|
||
**Step 2: Run, observe red on 1–4** (each for its own reason; report per-test).
|
||
|
||
**Step 3: Implement.** A small per-pass accumulator threaded through the pass:
|
||
|
||
```csharp
|
||
private sealed class ReadPassOutcome
|
||
{
|
||
public bool AnySuccess;
|
||
public DriverState WorstFailure = DriverState.Healthy; // Healthy = none recorded
|
||
public string? FirstError; // first failure is causal
|
||
public void RecordSuccess() => AnySuccess = true;
|
||
public void RecordFailure(DriverState state, string message)
|
||
{
|
||
if (Rank(state) > Rank(WorstFailure)) WorstFailure = state;
|
||
FirstError ??= message;
|
||
static int Rank(DriverState s) => s switch
|
||
{
|
||
DriverState.Faulted => 2, DriverState.Degraded => 1, _ => 0
|
||
};
|
||
}
|
||
}
|
||
```
|
||
|
||
- The three success sites (`:647`, `:828`, `:941`) call `outcome.RecordSuccess()` instead
|
||
of assigning `_health`.
|
||
- `RecordReadFailure` takes the outcome, records `(Degraded|Faulted, message)` into it
|
||
instead of assigning `_health`, and KEEPS its `_plcDead` / `MarkConnectionDeadIfFatal`
|
||
side effects immediate. It still returns the per-item status code.
|
||
- One `_health` write at the end of the `ReadAsync` try body (NOT `finally` — the OCE
|
||
arms at `:649`/`:801`/`:920` rethrow and must skip it):
|
||
|
||
```csharp
|
||
_health = new DriverHealth(
|
||
outcome.WorstFailure == DriverState.Healthy
|
||
? DriverState.Healthy : outcome.WorstFailure,
|
||
outcome.AnySuccess ? now : _health.LastSuccessfulRead,
|
||
outcome.WorstFailure == DriverState.Healthy ? null : outcome.FirstError);
|
||
```
|
||
|
||
- Untouched: the `:567` connect-failure early return; `HandlePollError` (`:1980`); the
|
||
entire write path; init/shutdown sites. `ReadSpanAsync` / `ReadLeftoversByMultiVarAsync`
|
||
gain an outcome parameter — signature changes, no logic movement.
|
||
|
||
**Step 4: Run the full S7 suite** (`dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.S7.Tests`)
|
||
— all green including the parity anchor `:202` (all-members-fail ⇒ Degraded both paths).
|
||
|
||
**Step 5: Falsify.** Via Edit: (a) make the end-of-pass write use the LAST recorded state
|
||
instead of the worst → tests 1/3/4 red; (b) gate `LastSuccessfulRead` advancement on the
|
||
state being Healthy → test 2 red. Fresh builds. Revert via Edit.
|
||
|
||
**Step 6: Commit** (explicit pathspec, message
|
||
`fix(s7): read-pass health is worst-state-wins — a mixed pass can no longer report Healthy`).
|
||
|
||
---
|
||
|
||
### Task 3: Fix stale `OTOPCUA_CONFIG_MODE` docs
|
||
|
||
**Classification:** trivial
|
||
**Estimated implement time:** ~2 min
|
||
**Parallelizable with:** Tasks 1, 2
|
||
|
||
**Files:**
|
||
- Modify: `docs/Configuration.md` (~line 253) — "Flip only the site nodes with
|
||
`OTOPCUA_CONFIG_MODE=FetchAndCache` at `docker compose up`" is stale
|
||
- Modify: `CLAUDE.md` — same sentence in the `ConfigSource`/`ConfigServe` section
|
||
- Check-and-fix if matched: `grep -rn "OTOPCUA_CONFIG_MODE" docs/ docker-dev/ CLAUDE.md`
|
||
|
||
**Spec.** Phase 4 made `FetchAndCache` mandatory on driver-only nodes
|
||
(`ConfigSourceOptionsValidator` fails host start on `Direct`), and the rig's four site
|
||
nodes are hardcoded `ConfigSource__Mode: "FetchAndCache"` in
|
||
`docker-dev/docker-compose.yml:572-579/691-696/787-792/858-863`. The env var appears
|
||
nowhere in the compose file; setting it is a no-op. Rewrite both sentences to state: the
|
||
site nodes run `FetchAndCache` unconditionally (mandatory for driver-only nodes since
|
||
Phase 4); central stays `Direct` via its admin role. No review (doc-only). Commit with
|
||
explicit pathspec, message `docs(config): OTOPCUA_CONFIG_MODE is gone — site nodes are hardcoded FetchAndCache (Phase 4)`.
|
||
|
||
---
|
||
|
||
### Task 4: Live gate — docker-dev first-POST-404 catch + no-body POST binding
|
||
|
||
**Classification:** live-gate (rig session step 1 — FRESH BOOT window)
|
||
**Estimated implement time:** ~10 min wall (mostly rig boot)
|
||
**Parallelizable with:** none (rig serial; runs after Tasks 1–3 land, images built from branch tip)
|
||
|
||
**Files:**
|
||
- Possibly modify: `docker-dev/deploy.sh` (the header NOTE about no-body binding),
|
||
`docker-dev/README.md` (§"First-POST 404" — mechanism note if caught)
|
||
- Evidence: write to the scratchpad; Task 8 folds it into the plan's Outcomes
|
||
|
||
**Procedure:**
|
||
1. Fresh rig: `docker compose -f docker-dev/docker-compose.yml down -v` then
|
||
`up -d --build` (build mandatory — the branch tip must be in the images).
|
||
2. IMMEDIATELY run `./docker-dev/deploy.sh` (no body). Its 2 s `/health/ready` poll rides
|
||
through Traefik during the boot window — every poll is access-logged with
|
||
`defaultmode=keep`, and an unmatched request shows an EMPTY `RouterName`.
|
||
3. After the deploy completes (or fails), capture
|
||
`docker compose -f docker-dev/docker-compose.yml logs traefik` and extract every
|
||
non-2xx access-log line from the boot window. **If any 404 appears:** name the
|
||
mechanism from `RouterName`/`ServiceName`/timing (the r2 elimination says: AdminUI
|
||
cannot 404 that path; backend-down is 503; so 404 = no router matched at that instant
|
||
— presumably the window before the dynamic-config file provider loads or before the
|
||
health-checked service has a server). Update `docker-dev/README.md` §"First-POST 404"
|
||
with the observed mechanism. **If it does not reproduce:** record that honestly —
|
||
"did not reproduce under the readiness gate; register item stays open" — do NOT
|
||
invent a mechanism.
|
||
4. No-body POST binding: with the rig ready, run `./docker-dev/deploy.sh` bare and record
|
||
the HTTP status + response body. Determine whether the minimal-API `DeployRequest?`
|
||
binds null (and what the endpoint then does) or 400s. Update the header NOTE in
|
||
`deploy.sh` from "unverified" to the observed behaviour, and `README.md` if wrong.
|
||
5. Leave the rig UP for Task 5.
|
||
|
||
Commit any `deploy.sh`/`README.md` edits (explicit pathspec). Report: the access-log
|
||
excerpt, the no-body verdict, and whether the 404 reproduced.
|
||
|
||
---
|
||
|
||
### Task 5: Live gate — FetchAndCache legs on the running rig
|
||
|
||
**Classification:** live-gate (rig session step 2)
|
||
**Estimated implement time:** ~8 min wall
|
||
**Parallelizable with:** none (rig serial, after Task 4)
|
||
|
||
**Procedure** (rig already up from Task 4; a deploy already ran):
|
||
1. **Fetch + verify:** `docker compose -f docker-dev/docker-compose.yml logs site-a-1 |
|
||
grep -E "Fetched \+ verified artifact"` — expect
|
||
`Fetched + verified artifact for {DeploymentId} from http://central-… ({Bytes} bytes).`
|
||
2. **Served (central side):** `logs central-1 central-2 | grep "Served artifact"` —
|
||
`Served artifact for {DeploymentId} ({Bytes} bytes, {Chunks} chunks)`.
|
||
3. **Applied-from-fetch + cached:** `logs site-a-1 | grep -E "applied fetched deployment|cached applied deployment"` — both lines must appear (the `fetched` variant, NOT the
|
||
Direct `applied deployment` form).
|
||
4. **Boot-from-cache:** `docker compose restart site-a-1`, wait for boot, then
|
||
`logs site-a-1 | grep "FetchAndCache boot"` — expect
|
||
`restored served state for deployment {Id} (rev {Rev}) from the LocalDb pointer.`
|
||
5. **Absence proof:** on site-a-1 logs, assert NO occurrence of `entering Stale`,
|
||
`ConfigDb back`, or the Direct-form `applied deployment {Id} (rev` (without
|
||
`fetched`). A FetchAndCache node must never reach Stale.
|
||
6. Evidence excerpts to the scratchpad for Task 8. Rig stays UP for Task 6.
|
||
|
||
No commit expected. If any leg fails, STOP and report — that is a real regression, not a
|
||
gate formality.
|
||
|
||
---
|
||
|
||
### Task 6: Live gate — alarm-bearing surgical ratchet
|
||
|
||
**Classification:** live-gate (rig session step 3)
|
||
**Estimated implement time:** ~12 min wall
|
||
**Parallelizable with:** none (rig serial, after Task 5)
|
||
|
||
**Context.** The surgical-deploys classifier routes an alarm-bearing raw-tag delta
|
||
(`Alarm != null` on EITHER side) to `Rebuild`; a whitelisted non-alarm attribute delta
|
||
(`Writable` among them) is surgical. Never observed live. The ONLY seam is the applier
|
||
summary line (`AddressSpaceApplier.cs:322-326`):
|
||
`applied plan (kind={Kind}, …, surgicalRawTags={SurgicalRaw}, …, rebuild={Rebuild})`.
|
||
|
||
**Procedure:**
|
||
1. Pick TWO alarm-free raw tags in the rig's seeded config (browse `/raw` via AdminUI at
|
||
`http://localhost:9200`, login disabled — or read the ConfigDb `Tag` rows). Call them
|
||
TagS (surgical lever) and TagA (alarm lever).
|
||
2. **Surgical control first:** toggle TagS's Access level (Read ↔ ReadWrite) — via the
|
||
RawTagModal in the browser, or a direct ConfigDb UPDATE of the tag row (the rig's DB
|
||
on `10.100.0.35,14330`; acceptable for a classifier gate, note which route was used).
|
||
Deploy via `./docker-dev/deploy.sh`. Expect on central-1:
|
||
`kind=AttributeOnly, … surgicalRawTags=1, … rebuild=False`.
|
||
3. **Alarm add ⇒ rebuild:** add `"alarm": {"alarmType": "OffNormalAlarm", "severity": 500}`
|
||
to TagA's `TagConfig` JSON (raw-JSON textarea — available because the seeded device is
|
||
a Galaxy driver, which has no typed editor — or the same direct-DB route). Deploy.
|
||
Expect `kind=Rebuild, … surgicalRawTags=0, … rebuild=True`.
|
||
4. **Alarm remove ⇒ rebuild:** remove the `alarm` object, deploy, expect `rebuild=True`
|
||
again (Alarm != null on the PREVIOUS side also disqualifies).
|
||
5. **Restore:** revert TagS's access level, deploy once more (surgical again — also
|
||
re-proves the ratchet is per-deploy, not latched).
|
||
6. Evidence: the four applier lines (grep `docker compose logs central-1 site-a-1 |
|
||
grep "applied plan (kind="` — note which nodes applied). Scratchpad for Task 8. Rig
|
||
stays UP for Task 7.
|
||
|
||
No commit expected. Failure of the expected routing = STOP and report.
|
||
|
||
---
|
||
|
||
### Task 7: Live gate — ACL-enabled enforcement on site-a (re-shaped cache)
|
||
|
||
**Classification:** live-gate (rig session step 4)
|
||
**Estimated implement time:** ~20 min wall
|
||
**Parallelizable with:** none (rig serial, after Task 6)
|
||
|
||
**Context.** `Security:Acl:Enabled` has never been exercised live since the r1/r2 cache
|
||
re-shape (per-key shards, generation stamp, Task 1's CAS loop). Site-a is the env-gated
|
||
demo pair: `OTOPCUA_ACL_ENABLED=true` reaches site-a-1/-2 only
|
||
(`docker-compose.yml:585/:702`); site-b stays off as the A/B control. Site-a OPC UA:
|
||
host ports 4842/4843; site-b: 4844/4845. Keys/grants reach driver-only site nodes ONLY
|
||
via the deployment artifact — every authoring step needs a deploy after it.
|
||
|
||
**Procedure:**
|
||
1. Restart site-a gated: `OTOPCUA_ACL_ENABLED=true docker compose -f docker-dev/docker-compose.yml up -d site-a-1 site-a-2`. Confirm the boot Warning on both:
|
||
`node ACL enforcement is ENABLED — EVERY data-plane operation … requires a matching NodeAcl grant`.
|
||
2. Mint a key: AdminUI `/apikeys` (Chrome automation; login disabled) — Name
|
||
`r3-acl-gate`, Principal `r3gate`. COPY the one-time `otua_…` credential immediately.
|
||
3. Author a grant: `/clusters/{site-a cluster id}/acls/new` — Principal `r3gate`, scope
|
||
picked via the AclScopePickerModal on a WRITABLE raw tag (or its device — inheritance
|
||
covers the tag), permissions: the Operator bundle (must include Browse/Read/
|
||
WriteOperate). Save; expect no "authorizes nobody" warning.
|
||
4. Deploy (`./docker-dev/deploy.sh`) so the key + grant ride the artifact to site-a.
|
||
Confirm on site-a-1: `data-plane authority generation {G} installed (… keys=…, grants=…)`.
|
||
5. **The load-bearing ALLOW:** Client.CLI write through the grant —
|
||
`dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- write -u opc.tcp://localhost:4842 -n "<granted raw tag NodeId>" -v <value> -U apikey -P otua_…`
|
||
Expect success (the session carries NO server-wide role; the grant alone authorizes —
|
||
a deny-only gate would pass against a build where the ACL is never consulted).
|
||
Note: the rig's Galaxy driver has no live gateway, so the DEVICE write may fail
|
||
downstream — the gate is that the request passes the ACL (not `BadUserAccessDenied`);
|
||
any downstream Bad status is fine and should be reported as such.
|
||
6. **Deny controls:** (a) same write WITHOUT credentials (anonymous) ⇒ denied; (b) a
|
||
read/browse on a node OUTSIDE the granted scope with the key ⇒ withheld (grep
|
||
site-a-1 for `AclGate: denied {Operation} on {NodeId} for principal r3gate`).
|
||
7. **Revocation reaches the artifact path:** disable the key on `/apikeys`, deploy,
|
||
confirm a NEW `data-plane authority generation {G+1} installed` on site-a-1, and the
|
||
same CLI write now fails auth (`ApiKeyAuth: denied — no enabled, unexpired key`).
|
||
8. **Teardown:** restore site-a ungated (`docker compose up -d site-a-1 site-a-2` without
|
||
the env), optionally delete the gate key/grant, and `docker compose down` the rig
|
||
unless Task 8 needs it. Evidence excerpts to the scratchpad.
|
||
|
||
No commit expected. If Chrome automation of the scope picker proves impractical, direct
|
||
ConfigDb inserts for `ApiKey`/`NodeAcl` rows are acceptable ONLY as a fallback — say so,
|
||
and keep the key's peppered HMAC consistent with the rig pepper
|
||
(`docker-dev-apikey-pepper-not-for-production`); minting via the UI is strongly preferred
|
||
because it exercises the real authoring path.
|
||
|
||
---
|
||
|
||
### Task 9: Roslyn evaluator racing test — remove the timing assumption
|
||
|
||
**Classification:** standard
|
||
**Estimated implement time:** ~5 min
|
||
**Parallelizable with:** Task 10 (disjoint files); independent of the rig tasks
|
||
|
||
**Origin:** ScadaBridge deferred-work register row 38, relayed cross-session 2026-08-18
|
||
(coordinated by the ScadaBridge session on Joe's behalf).
|
||
|
||
**Files:**
|
||
- Modify: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/RoslynVirtualTagEvaluatorTests.cs`
|
||
(`Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed`, ~:391 — renamed by
|
||
the implementation to `Evaluate_racing_ClearCompiledScripts_never_leaks_a_raw_disposed_failure`)
|
||
|
||
**Spec.** The test is load-fragile: it fails under full-solution parallel test load but
|
||
passes 3/3 in isolation. Its design assumes at most ONE `ClearCompiledScripts` lands
|
||
inside a single evaluation (so the product's one-shot retry suffices); under contention
|
||
two can land and the retry legitimately exhausts. Test-design issue, NOT a product bug.
|
||
Fix by removing the timing assumption while keeping the original intent (no
|
||
disposed-script failure ever ESCAPES the retry): either bound the clear cadence relative
|
||
to observed evaluation latency, or assert on retry-exhaustion behaviour instead of zero
|
||
failures — implementer picks whichever preserves intent with less machinery, documents
|
||
the choice in the test doc-comment, and demonstrates the test survives a
|
||
full-parallel-load run (run the containing suite alongside another heavy suite, or
|
||
repeat-run the suite ≥5×; report what was actually run).
|
||
|
||
---
|
||
|
||
### Task 10: LocalDb conditional registration + baseline + deregister self-heal
|
||
|
||
**Classification:** high-risk
|
||
**Estimated implement time:** ~8 min
|
||
**Parallelizable with:** Task 9 (disjoint files)
|
||
|
||
**Origin:** ScadaBridge deferred-work register row 39, relayed cross-session 2026-08-18.
|
||
ScadaBridge closed the same class with conditional registration + boot-time self-heal
|
||
(`Host/SiteLocalDbSetup.cs`, arch-review WP1.3/WP3.3); port that pattern here.
|
||
|
||
**Files:**
|
||
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSetup.cs` (`OnReady`,
|
||
registration at :63-66)
|
||
- Read for the predicate: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbRegistration.cs`
|
||
(`LocalDb:SyncListenPort`, `LocalDb:Replication:*` keys)
|
||
- Modify: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDbSetupTests.cs`
|
||
(the registered-table-set + ordering pins — update DELIBERATELY, say so in the diff)
|
||
- Modify: `CLAUDE.md` LocalDb section — the two bullets recording unconditional
|
||
registration as deliberate ("DeregisterReplicated exists and this repo deliberately
|
||
does not call it" / "Consequently, baselineExistingRows is not used either") must be
|
||
rewritten to record the new conditional shape and why it changed.
|
||
|
||
**Spec.**
|
||
- "Replication configured" = `PeerAddress` OR `ApiKey` present (OR, because only the
|
||
dialling half sets PeerAddress while the passive half carries just the key + listen
|
||
port — verify the exact key names against `LocalDbRegistration`).
|
||
- Configured ⇒ `RegisterReplicated(table, baselineExistingRows: true)` per table, in the
|
||
existing DDL→register order (the ordering is load-bearing and pinned) — seeds
|
||
row-versions at the LWW floor and flags snapshot resync, so late-enabled replication
|
||
on an established node converges on EXISTING rows, not just subsequent writes.
|
||
- NOT configured ⇒ `DeregisterReplicated` on every table at boot — idempotent self-heal:
|
||
drops stale triggers, prunes oplog/row-version rows, logs once at Information when it
|
||
actually cleaned something (and stays silent when there was nothing to clean).
|
||
- Registration stays per-table in a loop (0.3.0 made the snapshot flag per-table; the
|
||
loop preserves that granularity if the pin ever moves).
|
||
- **Pin note:** this repo is on `ZB.MOM.WW.LocalDb 0.2.1`, which has both
|
||
`baselineExistingRows` and `DeregisterReplicated`. Do NOT bump the pin unless a needed
|
||
API is genuinely absent at 0.2.1 — if it is, stop and report before bumping.
|
||
- **Traps (both hit by ScadaBridge):** the registered-table set is pinned by
|
||
`LocalDbSetupTests` — update the pins deliberately with the behaviour change named in
|
||
the test doc-comment; and deregistration must be pair-symmetric — the sync handshake
|
||
compares registered-table digests fail-closed, so flipping registration is a
|
||
both-nodes-together change. Record the pair-symmetry consequence in CLAUDE.md and in
|
||
the rollout note (docker-dev site-a has replication ON and was enabled from fresh, so
|
||
current impact is low; the change is correctness-before-someone-enables-late).
|
||
- Tests: pin the new predicate both ways (configured ⇒ registered with baseline;
|
||
unconfigured ⇒ deregistered/self-healed) and keep the DDL→register ordering pin green.
|
||
|
||
---
|
||
|
||
### Task 8: Close-out — sweep + Outcomes + register annotations
|
||
|
||
**Classification:** close-out (blockedBy: all)
|
||
**Estimated implement time:** ~15 min
|
||
|
||
1. Full build (`dotnet build ZB.MOM.WW.OtOpcUa.slnx`) — 0 errors; warning count vs r2's
|
||
904 explained.
|
||
2. Full unit sweep, every `*.Tests` project, per-project; reconcile the passed-count
|
||
delta against Tasks 1–2's added/changed tests exactly. DB suites must RUN (AdminUI
|
||
`0 skipped`, Configuration `0 skipped`).
|
||
3. Write `## Outcomes` on this plan: sweep record, per-task table, all four live-gate
|
||
evidence excerpts, falsification notes, and this round's register (Modbus + S7-write
|
||
asymmetry, TM/CT real-CPU, cross-process reap race carried; whatever Tasks 4–7 leave
|
||
open).
|
||
4. Annotate the r2 register in `2026-08-17-perf-followups-r2.md` append-only with
|
||
"Closed by <sha> (r3)" / "Gated live (r3)" sub-bullets.
|
||
5. Update the `.tasks.json` ledger; commit docs with explicit pathspec.
|
||
|
||
---
|
||
|
||
## Outcomes
|
||
|
||
*Written at close-out, 2026-08-18, on branch `perf/followups-r3`. All 10 tasks complete
|
||
(8 planned + Tasks 9/10 joined mid-round by amendment `088a1a1e`). This round is a
|
||
**follow-up round**: every code item it closes was opened by the round-2 register in
|
||
`2026-08-17-perf-followups-r2.md`, and that register is annotated in place with the closing
|
||
SHAs. Tasks 9 and 10 are **cross-repo** — rows 38 and 39 of the ScadaBridge deferred-work
|
||
register, relayed on Joe's behalf and closed here.*
|
||
|
||
### Sweep record
|
||
|
||
**The docker-dev rig was torn down BEFORE the sweep** (`docker compose -f
|
||
docker-dev/docker-compose.yml down`, 0 `otopcua-dev-*` containers remaining). This is not
|
||
housekeeping — a running rig un-skips env-gated tests and would have made the skip count
|
||
incomparable to the r2 baseline. It did: `Host.IntegrationTests` reports **229 / 0 / 2** with
|
||
the rig up and **226 / 0 / 5** with it down, the same 231 total.
|
||
|
||
`dotnet build ZB.MOM.WW.OtOpcUa.slnx --no-incremental` → **0 errors, 904 warnings** —
|
||
**identical to r2's 904**, not merely close. The non-incremental flag matters: an incremental
|
||
build reported 896 because up-to-date projects re-emit nothing. The two pre-existing families
|
||
are unchanged in kind and in count (`OTOPCUA0001` CapabilityInvoker analyzer on test call sites
|
||
1012, `xUnit1051` cancellation-token advice 696, plus the small CS tail). **No delta to
|
||
attribute** — this round's new test call sites happened to land in files already carrying the
|
||
same families, so nothing moved.
|
||
|
||
Full unit sweep — every `*.Tests` project, run per-project with `--no-build` after the single
|
||
solution build, against the tree at `837d8a17`. `*.IntegrationTests` are build-only by
|
||
convention (as in r2); `Host.IntegrationTests` is recorded **separately below** because r2's
|
||
baseline did not include it.
|
||
|
||
**43 / 43 test projects green.**
|
||
|
||
| | Round 2 | **This round** | Δ |
|
||
|---|---|---|---|
|
||
| Passed | 10 370 | **10 377** | **+7** |
|
||
| Failed | 0 | **0** | — |
|
||
| Skipped | 19 | **19** | — |
|
||
| **Total** | 10 389 | **10 396** | +7 |
|
||
|
||
The +7 reconciles **exactly** against the two suites this round added tests to, with nothing
|
||
unaccounted for — confirmed at class granularity, not just project granularity:
|
||
|
||
| Project | Round 2 | This round | Δ | Source |
|
||
|---|---|---|---|---|
|
||
| `OpcUaServer.Tests` | 481 | **482** | +1 | `SessionAclDecisionCacheTests` 17 → **18** (T1) |
|
||
| `Driver.S7.Tests` | 401 | **407** | +6 | `S7SpanReadParityTests` 13 → **19** (T2) |
|
||
|
||
⚠️ **The S7 +6 is the sum of three separate commits, not one addition** — `4c161181` (+3, the
|
||
worst-state-wins pins), `b35e423c` (+1, the observe-nothing guard the spec review found), and
|
||
`1765e300` (+2, the code-review round's severity-escalation and OCE-flush pins). T3 (docs) and
|
||
T4–T7 (live gates) added no tests by design.
|
||
|
||
**All 19 skips are the pre-existing env-gated ones, project for project** — `Core.Tests` (1),
|
||
`Driver.Galaxy.Tests` (8), `Driver.Historian.Gateway.Tests` (6, the `LiveIntegration` category
|
||
needing the `wonder-sql-vd03` VPN), `OpcUaServer.Tests` (1), `Runtime.Tests` (3). **This round
|
||
added no skips**, and `Runtime.Tests` is unchanged at 637 / 3 as expected.
|
||
|
||
The DB-backed suites ran **against a reachable** `10.100.0.35,14330` — executed, not skipped:
|
||
`AdminUI.Tests` **1540 passed / 0 skipped**, `Configuration.Tests` **190 passed / 0 skipped**.
|
||
|
||
#### `Host.IntegrationTests` (T9 + T10) — **226 passed / 0 failed / 5 skipped** (231 total)
|
||
|
||
Not part of the r2 baseline, so there is no measured predecessor. Derived from the diff against
|
||
the merge-base `49d05bc2`, the expected delta is **+9**, all plain `[Fact]` (no theories, so
|
||
attribute count equals test count): `LocalDbSetupTests` **7 → 15** (+8) and `LocalDbWiringTests`
|
||
**6 → 7** (+1); `RoslynVirtualTagEvaluatorTests` stays **20 → 20** because T9 reworked its
|
||
racing test *in place*. ⚠️ **The task brief's "LocalDbSetupTests 13 → 15" was an intermediate
|
||
reading** — 13 was the count after `3e62f161` and before the `df533916` review round; against
|
||
the merge-base it is 7 → 15. The 5 skips are the rig/fixture-gated
|
||
`DriverTestConnectE2eTests` (2, Modbus fixture) and `DriverProbeHandshakeE2eTests` (3) — three
|
||
of which run when the rig is up, which is exactly the teardown effect noted above.
|
||
|
||
#### ⚠️ One flake, isolated and cleared — `Client.CLI.Tests`
|
||
|
||
The first sweep pass reported `Client.CLI.Tests` **1 failed / 103 passed**:
|
||
`LoggerLifecycleTests.ConfigureLogging_DisposesPreviousLogger_BeforeReassigning`, asserting
|
||
`trackerSink.Disposed` was `False`. **Isolation re-runs cleared it: the single test alone
|
||
1/1 green, and the whole project green twice more (104 / 104 both times).**
|
||
|
||
The mechanism is a **pre-existing intra-assembly race on the process-global
|
||
`Serilog.Log.Logger` static**, not a regression: the test assigns `Log.Logger` at
|
||
`LoggerLifecycleTests.cs:26` and then asserts, at `:41`, that the command's `ConfigureLogging`
|
||
disposed *that* instance — but the class carries no `[Collection]` and no
|
||
`DisableTestParallelization`, so any concurrently-scheduled test whose command path also calls
|
||
`ConfigureLogging` (which does `Log.CloseAndFlush()` then reassigns) can swap the global out
|
||
from under it in that window. **No commit in this round touches `src/Client` or `tests/Client`**
|
||
(`git log 49d05bc2..HEAD -- src/Client tests/Client` is empty). Recorded here rather than in
|
||
the register, which the close-out brief scoped explicitly; it is a worthwhile one-line fix
|
||
(pin the class to a serial collection) whenever someone is next in that file.
|
||
|
||
### Per-task outcomes
|
||
|
||
| # | Task | Commit(s) | Outcome + deviation |
|
||
|---|---|---|---|
|
||
| 1 | ACL `InvalidateIfStale` — monotonic-max CAS loop | `441b34a3`, `f4b91dbb`, `e29fab19` | Done; closes the r2 register's deferred-publication item. ⚠️ **The plan's own 200-round `Barrier` harness was measured GREEN-ON-DEFECT** (~1-in-19k hit rate — `Barrier`'s post-phase release hands one racer a head start), so it was rebuilt as long-lived racers with a third-thread release store: 25–50 % of rounds red, falsification red at rounds 1–13. 10 000 rounds kept over the reviewer's suggested 2 000 after measuring a **1-in-3 vacuous pass at 2 000** under full-suite parallelism (misses burst-correlate). 18/18. |
|
||
| 2 | S7 read-pass health — worst-state-wins | `4c161181`, `b35e423c`, `1765e300` | Done; closes the r2 register's last-writer-wins item **for the S7 READ path only** (asymmetry recorded below). The spec review found a second defect in-round — an **empty pass that observed nothing must report nothing**, or it clears a live `Faulted`. The code review then found that the OCE arms skipped the end-of-pass write and froze `LastSuccessfulRead`; the coordinator ruled the OCE path must **flush** the recorded pass evidence via `finally`, which is what protects the watchdog contract. Five falsifications, each red on its own test. 19/19. |
|
||
| 3 | Stale `OTOPCUA_CONFIG_MODE` docs | `e9ba1741` | Done as specified; compose facts verified before editing. Historical plan docs deliberately untouched. Trivial ⇒ no review. |
|
||
| 4 | Live gate — first-POST 404 + no-body POST | `3f04a4d4` | **PASSED — and the 404 mechanism is now NAMED**, closing an r2 register item the gate was only expected to instrument. Detail below. |
|
||
| 5 | Live gate — FetchAndCache legs | *(none — gate only)* | **PASSED, all six legs.** Detail below. |
|
||
| 6 | Live gate — alarm-bearing surgical ratchet | *(none — gate only)* | **PASSED, all four routings.** Detail below. |
|
||
| 7 | Live gate — ACL-enabled enforcement | *(none — gate only)* | **PASSED**, with a topology adaptation and two non-blocking findings. Detail below. |
|
||
| 9 | Roslyn evaluator racing test — remove the timing assumption | `34829ea0`, `837d8a17` | Done. **Cross-repo: ScadaBridge deferred-work register row 38**, relayed 2026-08-18. Detail below. |
|
||
| 10 | LocalDb conditional registration + baseline + deregister self-heal | `3e62f161`, `df533916` | Done. **Cross-repo: ScadaBridge deferred-work register row 39.** No pin bump. **The scadaproj umbrella index was updated and pushed to `main` as `1145df6`.** Detail below. |
|
||
| 8 | Close-out — sweep + Outcomes + register annotations | *this commit* | Done. Rig torn down first, full build + sweep, this Outcomes section, and the append-only r2 register annotations. |
|
||
|
||
### Task 4 — the first-POST 404, mechanism CONFIRMED
|
||
|
||
**2026-08-18, local OrbStack docker-dev rig built from the branch tip. Two results, and the
|
||
second is the one that matters.**
|
||
|
||
- **The prescribed fresh-boot run did NOT reproduce the 404** — the entire Traefik access log
|
||
for the boot window was **3 lines, all 2xx**, every one carrying
|
||
`RouterName = otopcua-admin@file`. ⚠️ **And `deploy.sh` under `up -d` can never reproduce
|
||
it**: Compose blocks on the `depends_on: service_healthy` chain, so `up -d` returned ~30 s
|
||
*after* Traefik started and the first poll landed long after the window closed.
|
||
- **So the window was exercised directly** — rapid-poll (~10 ms) across
|
||
`docker restart otopcua-dev-traefik-1`, twice. Result: `000` (port unbound) → **`404` for
|
||
~65–160 ms** → `200`. Body is exactly `404 page not found` (Go's `http.NotFound`), and the
|
||
access-log signature is the discriminator the r2 round asked for: **`RouterName = "-"`,
|
||
`ServiceName = "-"`, `0ms`** — no router matched, no backend dialled.
|
||
- **NAMED MECHANISM: the window between Traefik binding the `web` entrypoint and loading the
|
||
dynamic file-provider config from `/etc/traefik/dynamic.yml`.** The listener accepts
|
||
connections before the router table exists.
|
||
- **The competing candidate is excluded on evidence, not argument** — a health-checked service
|
||
with no healthy server yet would be a *matched* router answering **503**; there are **zero
|
||
503s** anywhere in the session's access log.
|
||
- ⚠️ **Caveat now recorded in both `deploy.sh` and the README: not all 404s are
|
||
access-logged.** Probe 1's seven 404s produced **zero** log lines; probe 2's produced 4 of 7.
|
||
The accesslog handler is itself part of the dynamic chain that has not been built yet, so
|
||
**an empty access log does NOT prove the 404 did not happen.**
|
||
- **No-body POST binding — VERIFIED**, four request shapes (script stdin, no data flag at all,
|
||
explicit empty body, valid JSON), all reaching the handler. The minimal-API `DeployRequest?`
|
||
**binds null**; it does not 400 and does not 415. A clean post-seal run returned **HTTP 202
|
||
`Accepted`**, and the DB confirms the null took the fallback: `CreatedBy = rest-api`.
|
||
|
||
### Task 5 — FetchAndCache, all six legs
|
||
|
||
**PASSED. Two independent full cycles** (deploy `1e990163` rev `0b9289fb`, then the revert
|
||
deploy `da56012f` rev `315de861`), both sealed `acks=6`.
|
||
|
||
- **Fetch + SHA-verify:** all four site nodes —
|
||
`Fetched + verified artifact for 1e99016312004ceca5f73aaba2c899b1 from http://central-1:4055 (1584 bytes).`
|
||
The template only logs *after* the SHA-256 == RevisionHash check, so its presence is the
|
||
verify proof. The byte count tracked the composition (1584 → 1468 on revert).
|
||
- **Serve (central):** four `Served artifact for … (1584 bytes, 1 chunks)` on `central-1`.
|
||
- **Applied-from-fetch + cached:** the **`applied fetched deployment`** variant on every site
|
||
node — and the two `Direct` central nodes emitted the plain `applied deployment` form on the
|
||
*same* deploy, which is the falsification control that the two templates are distinguishable.
|
||
- **Boot-from-cache:** after `restart site-a-1`,
|
||
`FetchAndCache boot — restored served state for deployment 1e9901… (rev 0b9289fb…) from the LocalDb pointer.`
|
||
— the wanted variant, not the empty-cache or failed-pointer ones, matching what the node
|
||
cached 52 s earlier.
|
||
- **Absence proof** over the node's whole 302-line lifetime: `entering Stale` **0**, any
|
||
`stale` **0**, `ConfigDb back` **0**, Direct-form apply **0**, and `[ERR]`/`[WRN]`/`[FTL]`
|
||
**0**. ⚠️ The Direct-form check must anchor on `": applied deployment "` — a naive
|
||
`grep "applied deployment"` matches the *cache* template and reports a false positive.
|
||
- **Bonus:** a post-restart browse showed both namespaces live (`ns=2` raw + `ns=3` UNS)
|
||
including the temporary gate area — proof the node materialised the address space from the
|
||
artifact it fetched and restored, not from a stale cache.
|
||
- ⚠️ **Method note worth carrying:** **same-revision deploys short-circuit the fetch path**
|
||
entirely (`DriverHostActor.HandleDispatchFromSteady` immediate-ACK). A bare re-run of this
|
||
gate on unchanged config shows **zero fetch lines** and that is not a failure — force a new
|
||
revision first. The gate did so with a temporary `UnsArea` row, then reverted; the revert
|
||
deploy came back **byte-identical** to the pre-gate revision.
|
||
|
||
### Task 6 — the alarm-bearing surgical ratchet, four deploys
|
||
|
||
**PASSED. All four applier lines emitted identically on `central-1` and `central-2`**, every
|
||
deploy sealed 6/6:
|
||
|
||
| Step | Lever | Applier line | Verdict |
|
||
|---|---|---|---|
|
||
| 1 | `AccessLevel` ReadWrite → Read | `kind=AttributeOnly, changed=1, surgicalRawTags=1, rebuild=False` | surgical ✅ |
|
||
| 2 | alarm ADD | `kind=Rebuild, changed=1, surgicalRawTags=0, rebuild=True` | rebuild ✅ (`Alarm != null` current side) |
|
||
| 3 | alarm REMOVE | `kind=Rebuild, changed=1, surgicalRawTags=0, rebuild=True` | rebuild ✅ (**previous** side also disqualifies) |
|
||
| 4 | `AccessLevel` Read → ReadWrite | `kind=AttributeOnly, changed=1, surgicalRawTags=1, rebuild=False` | surgical ✅ — **the ratchet is per-deploy, not latched** |
|
||
|
||
- **Deviation, and why it is sound:** the rig's ConfigDb holds exactly **one** `Tag` row, so
|
||
TagS and TagA are the same tag exercised sequentially. Each step is an independent deploy
|
||
producing exactly **one** raw-tag delta over orthogonal fields, so every verdict is
|
||
unambiguous; a second tag would have added an *added-node* delta for no gain.
|
||
- **Proven at the artifact seam, not just the log:** each `Deployment.ArtifactBlob` was pulled,
|
||
gunzipped and semantically diffed — **each step's artifact differs by exactly the intended
|
||
lever and nothing else.**
|
||
- **Only the cluster-MAIN nodes emit an applier line.** Site nodes acked every deploy but
|
||
logged nothing: `DeploymentArtifact.ResolveClusterScope` self-scopes the fleet-wide artifact,
|
||
the site plan is empty, and `Apply` early-returns on `plan.IsEmpty` before the summary log.
|
||
Confirmed they *did* process each artifact via their authority-generation lines.
|
||
- ⚠️ **A live-gate fact worth knowing for every future rig gate: the revision hash can never
|
||
return to its baseline after any `UPDATE`.** `ConfigComposer` snapshots the SQL Server
|
||
`rowversion` column into the artifact, and `rowversion` increments on every `UPDATE` including
|
||
one that writes back an identical value. A byte-diff confirmed `RowVersion` was the **only**
|
||
difference in the entire 4922-byte artifact. Semantic restore was verified instead (config
|
||
hash back to `7F75AB45…`, rowcount unchanged).
|
||
|
||
### Task 7 — ACL-enabled enforcement, first live run since the cache re-shape
|
||
|
||
**PASSED**, with a forced topology adaptation and two findings.
|
||
|
||
- ⚠️ **The planned gated node was not executable.** `site-a` serves **containers only** — the
|
||
rig's single tag belongs to cluster **MAIN**, so `site-a-1`'s authority generation always
|
||
reports `keys=0, grants=0, resolvedScopes=0` and there is no writable value node to authorize.
|
||
The gate moved to **`central-1`** (fused `admin,driver,cluster-MAIN`, serving
|
||
`ns=2;s=pymodbus/plc/HR200`) via a **scratchpad-only compose overlay** — the repo compose
|
||
file was never edited — with **`central-2` as the ungated A/B control**. Consequence recorded:
|
||
central-1 is a ConfigDb node so *key* revocation is immediate rather than next-deployment;
|
||
**grants still arrive only via the artifact**, so the code actually under test is exercised
|
||
as intended.
|
||
- **Authoring fell back to direct ConfigDb INSERT** (the Chrome extension was not connected).
|
||
`ApiKeySecret.Generate` was reproduced from source and the pepper read live off the container;
|
||
the construction is proven correct because the server accepted the key.
|
||
- **The ALLOW leg is genuinely load-bearing, because the ungated baseline was captured first.**
|
||
The *identical* key and call on the *ungated* node returned `0x801F0000` — the `Tag`-scoped
|
||
grant confers no role, so `ApiKeyInterimRoleMapper` leaves the session with `roles []` and the
|
||
server-wide gate denies. With the gate on, the same call returned **`Write successful`**.
|
||
That is the live demonstration that **the ACL decision REPLACES the role check** rather than
|
||
chaining with it — a deny-only gate would have passed against a build that never consults the
|
||
ACL.
|
||
- **Denials fail closed:** anonymous write and read both `0x801F0000` with
|
||
`AclGate: Read withheld 1 of 1 node(s) for principal anonymous`; out-of-scope browse refused;
|
||
the ungated control node still open on the same NodeId.
|
||
- **Revocation was split into its two distinct code paths, and only the first tests this
|
||
round's fix.** *Grant removal* → generation **1 → 2**, `grants=0, resolvedScopes=0`, a
|
||
deliberately **primed** decision cache discarded, and the denial arrives while
|
||
`ApiKeyAuth: granted` still logs — that is the per-key-shard/CAS-loop path. *Key disable* →
|
||
`0x80210000` `BadIdentityTokenRejected` and
|
||
`ApiKeyAuth: denied — no enabled, unexpired key with id …` — a different status, because the
|
||
session never activates and never reaches the gate.
|
||
- **Restore verified by hash:** the post-task revision returned to the exact pre-task
|
||
`a3a9061bbf50…`, with 0 `ApiKey` / 0 `NodeAcl` rows and the ENABLED warning gone.
|
||
|
||
### Task 9 — the racing test, and an honest disclosure
|
||
|
||
Renamed to `Evaluate_racing_ClearCompiledScripts_never_leaks_a_raw_disposed_failure` and
|
||
re-shaped around **three-bucket classification** (leaked ⇒ fail on first; exhausted and starved
|
||
each bounded at `max(evals/10, 8)`) with duration-bounded loops (10 s **or** 400 iterations).
|
||
|
||
- ⚠️ **The real load failure was not the one the ticket described.** It was the test's own 45 s
|
||
`WaitAsync` budget expiring under recompile pressure, not the retry legitimately exhausting.
|
||
Old shape: **RED 4/4** at 4-way concurrency. New shape: **GREEN 4/4, then 8/8**.
|
||
- The review found a further flake vector — `TimedScriptEvaluator`'s 5 s clock starts at
|
||
**enqueue**, so a starved evaluation can time out without any product fault. Closed and
|
||
falsified with a 12 ms-budget probe (old classification RED, new GREEN at a 0.11 % starve rate).
|
||
- Failure-reason literals were promoted to a shared `ScriptEvalFailureReasons` in Commons, used
|
||
by both evaluators and the test.
|
||
- ⚠️ **Disclosed honestly: the disposed race does not reproduce on this arm64 macOS box at
|
||
all.** The raw-escape assertion is the load-independent guarantee; the exhaustion/starvation
|
||
bounds are calibrated loose (see the register).
|
||
|
||
### Task 10 — LocalDb registration now follows replication config
|
||
|
||
Ported from ScadaBridge's WP1.3/WP3.3 pattern. Predicate = **`PeerAddress` OR `ApiKey`**
|
||
(`SyncListenPort` deliberately excluded — the passive endpoint fails closed on the key).
|
||
Configured ⇒ `RegisterReplicated(table, baselineExistingRows: true)` per table in the pinned
|
||
DDL→register order; not configured ⇒ `DeregisterReplicated` on every table at boot as an
|
||
idempotent self-heal. The `CLAUDE.md` bullets that recorded the *unconditional* shape as
|
||
deliberate were rewritten in the same commit.
|
||
|
||
- **No pin bump.** The 0.2.1 APIs were verified by reflection **and** a live probe (baseline
|
||
seeds `hlc=0`; `DeregisterReplicated` returns `bool`) rather than assumed from the changelog.
|
||
- **Two vacuous-fixture bugs were surfaced and fixed** in passing (`LocalDbPairHarness`, and an
|
||
`AlarmSfLegacyMigrator` two-config split) — fixtures that would have let the new predicate
|
||
pass without being exercised.
|
||
- **Pair-symmetry is the operational trap**, recorded in `CLAUDE.md` and the runbook: the sync
|
||
handshake compares registered-table digests fail-closed, so flipping registration is a
|
||
**both-nodes-together** change.
|
||
- The scadaproj umbrella index was updated and **pushed to `main` as `1145df6`**, per the
|
||
standing cross-repo rule.
|
||
|
||
### Review chain
|
||
|
||
9 implementers; **12 review verdicts** across the 4 reviewed code tasks (T1 spec + code +
|
||
delta re-review; T2 spec + code + delta re-review; T9 spec + code + delta re-review; T10 spec +
|
||
code + delta re-review); **4 fix rounds**; **2 CHANGES REQUESTED** (T2, T9). T3 was trivial
|
||
(doc-only ⇒ no review) and T4–T7 are live gates with no code to review. **Zero findings papered
|
||
over**; one coordinator ruling (T2's OCE flush).
|
||
|
||
Three **evidence-hygiene** lessons from this round, all worth carrying forward:
|
||
|
||
- ⚠️ **A race harness can be green-on-defect, and you must measure its hit rate to know.**
|
||
T1's plan-supplied `Barrier` harness "passed" at ~1-in-19 000 — it looked like a working test.
|
||
Only measuring the hit rate exposed that `Barrier`'s post-phase release hands one racer a head
|
||
start. **Round count is a function of the measured hit rate under full-suite parallelism**, not
|
||
a round number: 2 000 rounds was measured vacuous 1 time in 3 because misses burst-correlate.
|
||
- ⚠️ **Tear the rig down before a comparison sweep.** A running rig silently un-skips env-gated
|
||
tests; the r2 skip count is only comparable because the rig was down for both.
|
||
- ⚠️ **Name the version of a count you are quoting.** The "13 → 15" in this round's own close-out
|
||
brief was a mid-round reading; against the merge-base it is 7 → 15. Intermediate counts drift
|
||
the moment a review round lands.
|
||
|
||
### Follow-up register (opened by this round)
|
||
|
||
Only genuinely **new or still-open** items. Everything this round closed is annotated on the
|
||
round-2 register. **None of these blocks the branch.**
|
||
|
||
**ACL**
|
||
|
||
- ⚠️ **Browse transit stops below the realm root — a grant-holding client cannot browse down to
|
||
its own granted tag.** With the gate on, the key browses `ns=2;s=pymodbus` and
|
||
`ns=2;s=pymodbus/plc` fine but is refused `0x801F0000` on `ns=2;s=OtOpcUa` and on a recursive
|
||
browse from the Objects root. `PermissionTrieNode.BrowseTransitPrincipals` marks the ancestors
|
||
lying on the tag's **RawPath**; the synthetic realm root sits *above* the RawPath space and is
|
||
not in the trie at all. It **fails closed** (not a security hole) and blocked no leg, but it
|
||
would make an ACL-enabled deployment look broken to any browse-driven client — including the
|
||
Client.CLI's own default recursive browse. Discovered by the T7 live gate. **RECOMMEND filing a
|
||
Gitea issue before anyone enables `Security:Acl:Enabled` in production.** Likely fix: mark the
|
||
realm-root container browse-transit whenever any grant resolves beneath it.
|
||
- **Closed by `89630ee4` + `ff9d617a` (branch `fix/acl-anchor-browse-transit`, 2026-08-18)** — no
|
||
Gitea issue needed; fixed directly. The actual mechanism was one step off the guess above: the
|
||
trie's zero-segment realm-root branch already existed and was correct — nothing ever *produced*
|
||
that scope, because the shared anchor's identifier (`s=OtOpcUa`, now `V3NodeIds.SharedRootFolderId`)
|
||
split into a one-segment scope no grant path contains. `DataPlaneAuthorizer.BuildScope` now maps
|
||
the anchor (realm-qualified: Raw namespace only) onto the zero-segment scope, and `Evaluate` adds
|
||
a Browse-only second walk against the zero-segment **Uns** scope — the single anchor fronts both
|
||
trees, so a UNS-only grant holder must also get through. 11 new tests (3 trie pins, 7 authorizer
|
||
incl. falsification of the OR-leg and of the realm qualification, 1 e2e browse-succeeds-and-still-
|
||
filters through a booted server). Full high-risk chain: spec ✅, code CHANGES REQUESTED (stale
|
||
"two namespace root folders" XML remark) → fixed → delta re-APPROVED.
|
||
- **`ns=0` standard nodes are ungated — the boot warning overstates.** With the gate on, both
|
||
the key and anonymous read `i=2256` and browse `i=2253` successfully: those are served by the
|
||
SDK's `CoreNodeManager` and never reach `DataPlaneAuthorizer`. Defensible (standard server
|
||
metadata, no process data), but "A principal with no grant can connect and see **nothing**"
|
||
is not literally true. Informational — a doc softening or a deliberate decision to gate ns=0.
|
||
|
||
**S7**
|
||
|
||
- **`_health` remains a plain field, versus Modbus's `Volatile.Read`/`Volatile.Write` idiom.**
|
||
Pre-existing, but **more of an outlier now** that the read pass consolidates to a single
|
||
end-of-pass write — the one place an alignment would be cheap and obvious. Follow-up
|
||
alignment, not a correctness claim.
|
||
- **Worst-state-wins landed for the S7 READ pass only.** Modbus (all paths) and the S7 **WRITE**
|
||
path keep last-writer-wins. **Deliberate scope** — both were named out of scope in this plan
|
||
because fixing them here doubles the blast radius of a watchdog-adjacent change. Recorded as a
|
||
known asymmetry so the next reader does not mistake it for an oversight.
|
||
|
||
**LocalDb**
|
||
|
||
- **`ZB.MOM.WW.LocalDb 0.3.1` is published to the Gitea feed** (per-table snapshot flags + a sync
|
||
fault-teardown fix). **This repo deliberately stays on `0.2.1`** — the 0.2.1 APIs cover
|
||
everything Task 10 needed, verified by probe. The per-**database** snapshot caveat is documented;
|
||
0.3.x makes it per-table. ⚠️ A bump is a **both-nodes-together** change (bookkeeping schema
|
||
v2 → v3 is one-way), so it wants a planned window, not an incidental pin edit.
|
||
|
||
**FetchAndCache**
|
||
|
||
- **`central-2`'s serve path was never exercised live.** All four site nodes resolved
|
||
`central-1:4055` first and stopped there — the documented walk-in-order, stop-at-first-success
|
||
behaviour, so central-2 served nothing. Not a fault; simply an untested half of the pair.
|
||
|
||
**Scripting / test calibration**
|
||
|
||
- **The disposed race does not reproduce on this arm64 macOS box at all.** The raw-escape
|
||
assertion is the load-independent guarantee and holds regardless; the **exhaustion and
|
||
starvation bounds are calibrated loose** on a machine that cannot produce the phenomenon they
|
||
bound. They would benefit from a Linux-CI observation to tighten.
|
||
|
||
**Test reliability**
|
||
|
||
- **`Client.CLI.Tests` `LoggerLifecycleTests.ConfigureLogging_DisposesPreviousLogger_BeforeReassigning`
|
||
is flaky under intra-assembly parallelism** — observed once in the r3 close-out sweep, green in
|
||
isolation (1/1) and on two whole-project re-runs (104/104). Mechanism: a pre-existing race on the
|
||
process-global `Serilog.Log.Logger` — the class carries no `[Collection]`, so a concurrent test's
|
||
`ConfigureLogging` (`CloseAndFlush` + reassign) can swap the global between arrange and assert.
|
||
No r3 commit touches `src/Client` or `tests/Client`. One-line fix when picked up: pin the class
|
||
to a serial collection.
|
||
|
||
**Carried from round 2, still open**
|
||
|
||
- **TM/CT decode at a non-zero device number still needs a real S7 CPU.** The test exists and the
|
||
env var (`S7_TIMER_NUMBER_ADDRESSING=1`) is ready; python-snap7 resolves TM/CT as a BIT address
|
||
and cannot serve it.
|
||
- **The cross-*process* plant-vs-reap race on the shared SQL server is accepted.** The 24 h age
|
||
floor is the mitigation; a lock table on a shared dev server costs more than it buys.
|
||
|
||
### Round-2 register items CLOSED by this round
|
||
|
||
- **ACL single-shot CAS deferred publication** → closed by **`441b34a3`** (T1).
|
||
- **S7 per-item decode health last-writer-wins** → closed by **`4c161181` + `b35e423c` +
|
||
`1765e300`** (T2), for the **read path**; the write-path and Modbus asymmetry is recorded above.
|
||
- **docker-dev first-POST 404** → **mechanism NAMED** by **`3f04a4d4`** (T4), and the **no-body
|
||
POST binding VERIFIED** by the same commit.
|