dohertj2/lmxopcua
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(plans): 1A.2 + 1B.2 close-out — register items 1-2 CLOSED, defect log (3), carry-forwards, decisions

Browse Source
This commit is contained in:
Joseph Doherty
2026-08-06 06:58:34 -04:00
parent 86633f140e
commit fbf7b70847
2 changed files with 262 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md
+36 -6
View File
@@ -23,8 +23,8 @@ posture).
| Phase | Scope | Branch | Status | PR / merge SHA | Live gate |
|---|---|---|---|---|---|
| 0 | Proto + mappers + goldens; both auth interceptors; `IMeshPskProvider`; options/validators (+`Grpc` value, SplitTopology relaxation); rig dark config; stale-comment fixes; FleetStatus issue filed | `feat/grpc-phase0-contract-auth` | **MERGED to `master`** @ `10b6d209` (2026-08-06) — all 6 tasks complete @ `23555ed0`; every task spec ✅ + code ✅ (Batch B on Opus with a closed fix loop); issues #542 + #543 filed | `10b6d209` | **PASS 4/4** |
| 1A | Central→node command plane (`NodeCommandService` on the Telemetry listener; `MeshGrpcCommandDialer`; third mode branch in `CentralCommunicationActor`) — **worktree A** | `feat/grpc-node-command` | NOT STARTED | — | PENDING |
| 1B | Node→central ack plane (`CentralCommandService` on the ConfigServe listener; `GrpcApplyAckSender`; third branch in `NodeCommunicationActor`) — **worktree B**, rebases on 1A | `feat/grpc-central-ack` | NOT STARTED | — | PENDING |
| 1A | Central→node command plane (`NodeCommandService` on the Telemetry listener; `MeshGrpcCommandDialer`; third mode branch in `CentralCommunicationActor`) — **worktree A** | `feat/grpc-node-command` | **1A.1 ✅ `64774652` + 1A.2 ✅ `6a862011` (+fixes `48dc2120`/`86633f14`)** — all reviews spec ✅ + code ✅ (1A.2 on Opus, fix loop closed); 1A.3 gate next | — | PENDING |
| 1B | Node→central ack plane (`CentralCommandService` on the ConfigServe listener; `GrpcApplyAckSender`; third branch in `NodeCommunicationActor`) — **worktree B**, rebases on 1A | `feat/grpc-central-ack` | **1B.1 ✅ `14dd07d2` + 1B.2 ✅ `a5a789d4` (+fixes `e54711a2`/`4d607575`)** — all reviews ✅ (1B.2 on Opus: 1 Critical found+fixed, see defect log); awaiting 1A merge → rebase → 1B.3 gate | — | PENDING |
| 2 | Baseline reproduction on pre-flip build, then full-rig cutover (`OTOPCUA_MESH_MODE` default → `Grpc`) + soak + kill drills | (rig commit) | NOT STARTED | — | PENDING |
| 3 | Delete ClusterClient (code/HOCON/config/tests); SplitTopology requires `Grpc`; grep-gates; docs sweep incl. scadaproj index | `feat/grpc-phase3-deletion` | NOT STARTED | — | — |
| 4 | Terminal live gate on the post-deletion image (8 checks, provenance-guarded) | (on `master`) | NOT STARTED | — | PENDING |
@@ -46,11 +46,14 @@ phase DoD — PR + merge are user-owned.**
| 2026-08-06 | No transport interface — third mode branch inside the two choke-point actors (ScadaBridge §7.1 economy) | design §4.3 |
| 2026-08-06 | Coexistence granularity: central's mode governs commands, each node's mode governs its acks (per-pair flips possible — better than ScadaBridge's central-wide flag) | design §3.6 |
| 2026-08-06 | `AlarmCommandMessage.operation` is a proto `string`, NOT an enum — the CLR `AlarmCommand.Operation` is an unconstrained string, and an enum would silently decode unknown operations to `UNSPECIFIED` (the drop class this migration removes). Plan T0.1's "AlarmCommand verb" enum wording was a plan defect; only `ApplyOutcome` carries the enum-0 rule. T0.2's goldens must pin pass-through of an unrecognized operation string | proto comment + this log (T0.1 review) |
| 2026-08-06 | **Gate follows the endpoint, not the mode** (1A.2 I-2): known-cluster set published on every contact load in all three modes, from ALL `ClusterNode` rows; fan set unchanged. Survives dark-switch rollback with nothing for 1B to remember | tracking §register refinement + actor docs |
| 2026-08-06 | `CentralCommandService` mapped by PORT, not mode-gated (1B.1): a mode flip on a running fleet must not need a central restart before acks flow; matches every sibling service | `Program.cs` comment + this log |
| 2026-08-06 | Ack deadline is PER ENDPOINT, not per walk; the walk is at-least-once on the deadline path, absorbed idempotently by the coordinator's NodeId-keyed upsert | `MeshTransportOptions` docs + defect log #1 |
| 2026-08-06 | `correlation_id` lives on the envelope AND as inner fields on the three commands whose CLR record carries one — each command round-trips independently in the T0.2 mapper; the envelope covers `AlarmCommand` (no CLR correlation member). For the three, the inner field is authoritative on decode | proto comment + this log (T0.1 review) |
## Phase 1A security register (from the Batch B Opus code review — HARD GATE: all four close BEFORE `CentralCommandService` is mapped)
1. **Known-cluster-set membership gate in `TryGetPsk`** — central rebuilds its cluster set from
1. **CLOSED (1A.2 @ `48dc2120`, refined — see below).** Known-cluster-set membership gate in `TryGetPsk` — central rebuilds its cluster set from
`ClusterNode` rows every `ContactRefreshSeconds`; refuse a cluster id outside that set in
O(1) with no store touch. Closes BOTH open review findings at once: (a) the timing oracle
that currently defeats the uniform-denial anti-enumeration text (unknown cluster pays a
@@ -61,13 +64,33 @@ phase DoD — PR + merge are user-owned.**
audit singleton, attacker-chosen `Target` values, before any credential check). Namespace
confinement (`OT-MESH-PSK-` prefix + `SecretName` allow-list) means no disclosure risk —
this is flood/oracle only.
2. **`bool Matches(string clusterId, string presented)` on `IMeshPskProvider`** — keep
2. **CLOSED (1A.2 @ `14d21496`).** `bool Matches(string clusterId, string presented)` on `IMeshPskProvider` — keep
`FixedTimeEquals` inside the provider so a future verify-side caller cannot write `==` and
have it compile. Do before 1B adds the dial-side caller.
3. **DI-shape pin test** — `IMeshPskProvider` is `hasAdmin`-only and its pairing with the
central interceptor is by convention; a driver-only node injecting it throws on first call,
not at boot. Add a Host.IntegrationTests check (admin host resolves it; driver-only host
does NOT) when Phase 1 maps the services.
**Refinement on item 1 (decision, 2026-08-06):** the gate follows the ENDPOINT, not the mode —
published on every contact load in every transport mode (the interceptor and service are
port-mapped, not mode-gated, so a dark-switch rollback must not deactivate the gate), and
built from ALL `ClusterNode` rows (gate answers "does central know this cluster"; the fan set
stays `Enabled && !MaintenanceMode`, so gate ⊇ fan — this also dissolved the
maintenance/ack-auth coupling). Pinned by the every-mode Theory + gate-⊋-fan fact + the Dps
no-ClusterClient guard pin (`86633f14`). Residuals accepted + visible: gate is fail-open until
the first successful contact load (and stays stale if central boots with SQL down — warned
every tick); gate derives from the node table, so a `ServerCluster` with zero rows is outside
it.
**Follow-ups from the 1A.2/1B.2 Opus reviews (tracked, not blocking):** per-tick
`InvalidateAll` puts a cold-cache 3 s store read on the inbound auth path — MEASURE on the
1B.3 gate (ack burst inside ~3 s after a refresh tick), then consider diff-based
invalidation; the command plane is h2c-only so the per-cluster PSK crosses the network in
cleartext on a write surface — precedent-consistent (all sibling gRPC planes), record as
accepted risk + TLS-option issue (queued below); dialer minors (Lazy channel wrapper,
retired-endpoint eviction, gRPC deadline vs token cancellation, pre-existing
`LoadContactsFromDb` Task.Run(ct) pattern).
4. **Integration gate over real Kestrel h2c** — both new interceptors have unit coverage only;
the one-public-ctor reflection test approximates but cannot prove the interceptor is
actually invoked. Wire-probe in the 1A/1B live gates (already in the live-gate tables).
@@ -104,7 +127,9 @@ Defects found during execution (live-gate or review), with root cause + pinned t
| # | Phase | Defect | Root cause | Fix / pin |
|---|---|---|---|---|
| — | — | *(none yet)* | | |
| 1 | 1B.2 | **Failover silently defeated on a hanging central** (CRITICAL, caught by Opus code review): the ack deadline was one absolute `DateTime` shared across the endpoint walk — a hung (not refused) central-1 burned the whole budget and central-2 was dialed with an expired deadline | deadline hoisted outside the `foreach`; a kill-style drill can't see it (refused = fast-fail), only a hang discriminates | `e54711a2`; pinned by `Each_endpoint_gets_a_FRESH_deadline` (mutation-verified); 1B.3 gains a `docker pause` hang drill |
| 2 | 1A.2 | **Dark-switch rollback deactivated the known-cluster gate** (Opus review, I-2): gate published only under `Mode=Grpc` while the interceptor + service are port-mapped in every mode — a rollback to ClusterClient/Dps would silently restore the timing oracle + pre-auth flood | gate activation coupled to transport mode instead of the endpoint it protects | `48dc2120` — gate follows the endpoint; pinned by the every-mode Theory + Dps guard pin `86633f14` |
| 3 | 1A.2 | **Fan parked the actor mailbox** (self-caught by the implementer): dialing from the receive ran `TryGetPsk`'s cold-cache 3 s block pre-await (cold after EVERY refresh due to `InvalidateAll`), stalling deployment acks behind the fan | async methods run synchronously to the first await | `Task.Run` hop (token None — a canceled scheduling would read as a DB failure via PipeTo); pinned red-first by `A_slow_dial_does_not_park_the_actors_mailbox` |
## Carry-forward register
@@ -112,13 +137,18 @@ Checks a gate could not drive, named + carried to the next gate (proportionate-g
| Item | Deferred from | Proven instead by | Closed in |
|---|---|---|---|
| — | — | *(none yet)* | |
| Hang-style central failover drill (`docker pause`, not `docker stop` — only a hang discriminates the per-endpoint-deadline fix) | 1B.2 review | unit fact `Each_endpoint_gets_a_FRESH_deadline` | 1B.3 gate (MUST include) |
| I-4 measurement: ack burst within ~3 s after a contact-refresh tick (cold PSK cache on the inbound auth path) | 1A.2 review | bounded 3 s store read + `Lazy` stampede suppression | 1B.3 gate (observe) |
| `TwoNodeClusterHarness` logs the I-1 null-provider Error at every spawn (harness registers no `IMeshPskProvider`) — EXPECTED, not a rig fault | 1A.2 re-review | production pairing is `hasAdmin`-unconditional in one block | note for 1B.3/Phase 2 readers |
| Register item 3 (DI role-scoping pin: admin resolves provider+dialer, driver-only does NOT) — container-activation test exists, role-scoping pin outstanding | 1A.2 | `It_activates_from_the_container_despite_the_optional_client_factory_parameter` | Phase 2 (or 1B.3) |
## Issues filed by this program
| Issue | What | Status |
|---|---|---|
| [lmxopcua#542](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/542) | `FleetStatusBroadcaster` derives "fleet" from central's own gossip ring — silently central-pair-blind since mesh Phase 6; pre-existing, transport-independent, out of this program's scope | FILED 2026-08-06 |
| *(to file in 1A.3)* | `ConfigPublishCoordinatorTimeoutTests.DeadlineElapsed_for_current_deployment_marks_TimedOut` is racy on its own terms (150 ms wall-clock deadline; row inserted after spawn) — flaked 2/~11 full runs once 1A.2 grew the suite; green at base. Fix the test (inject clock / raise deadline) | to file |
| *(to file in 1A.3)* | Mesh command/ack planes are h2c-only — the per-cluster PSK crosses the network in cleartext on a write surface (precedent: every sibling gRPC plane; trusted-site-link assumption). Decide: accepted risk, or a `UseTls`/scheme option | to file |
| [lmxopcua#543](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/543) | The three pre-existing auth interceptors (`TelemetryStream`/`ConfigServe`/`LocalDbSync`) treat a whitespace-only key as SET (`IsNullOrEmpty`), so a whitespace `Telemetry:ApiKey` etc. opens the endpoint to a whitespace bearer; the two new mesh interceptors use `IsNullOrWhiteSpace`. Backport is a behaviour change to shipped endpoints — own issue, out of this program's scope | FILED 2026-08-06 |
## Known constraints carried from the research (do not rediscover)
docs/plans/2026-08-06-clusterclient-to-grpc.md.tasks.json
+226 -17
View File
@@ -3,22 +3,231 @@
"executionModel": "opus-subagents; sequential within a worktree, 1A and 1B concurrent in separate worktrees (git worktree add ../OtOpcUa-1B feat/grpc-central-ack); 1A merges first, 1B rebases; stop at each phase DoD (PR/merge user-owned)",
"worktrees": {},
"tasks": [
{"id": 0, "subject": "Task 0.1: mesh_transport.proto + contract lock", "phase": "0", "branch": "feat/grpc-phase0-contract-auth", "classification": "standard", "parallelizableWith": ["0.3 (batch, same agent)"], "status": "completed", "commit": "c0ccfa27", "review": "spec ✅ + code ✅"},
{"id": 1, "subject": "Task 0.2: DTO mapper + round-trip goldens", "phase": "0", "branch": "feat/grpc-phase0-contract-auth", "classification": "high-risk", "parallelizableWith": [], "status": "completed", "blockedBy": [0], "commit": "545413a8", "review": "spec ✅ + code ✅ (serial)"},
{"id": 2, "subject": "Task 0.3: MeshTransportOptions + validators + SplitTopology relaxation", "phase": "0", "branch": "feat/grpc-phase0-contract-auth", "classification": "standard", "parallelizableWith": ["0.1 (batch, same agent)"], "status": "completed", "commit": "aa21f61f", "review": "spec ✅ + code ✅"},
{"id": 3, "subject": "Task 0.4: node-side MeshCommandAuthInterceptor", "phase": "0", "branch": "feat/grpc-phase0-contract-auth", "classification": "high-risk", "parallelizableWith": ["0.5 (batch, same agent)"], "status": "completed", "blockedBy": [2], "commit": "fbda015c + fixes 24e55d33/a4631410", "review": "spec ✅ + code ✅ (opus, fix loop closed)"},
{"id": 4, "subject": "Task 0.5: IMeshPskProvider + CentralCommandAuthInterceptor", "phase": "0", "branch": "feat/grpc-phase0-contract-auth", "classification": "high-risk", "parallelizableWith": ["0.4 (batch, same agent)"], "status": "completed", "blockedBy": [2], "commit": "68a416e4 + fixes 24e55d33/a4631410", "review": "spec ✅ + code ✅ (opus, fix loop closed); 4 items gated on Phase 1A — see tracking §Phase 1A security register"},
{"id": 5, "subject": "Task 0.6: rig config + FleetStatus issue + docs stub + Phase 0 gate", "phase": "0", "branch": "feat/grpc-phase0-contract-auth", "classification": "small", "parallelizableWith": [], "status": "completed", "blockedBy": [0, 1, 2, 3, 4], "commit": "23555ed0", "review": "code ✅", "gate": "PASS 4/4; issues #542 #543 filed"},
{"id": 6, "subject": "Task 1A.1: NodeCommandGrpcService on the telemetry listener", "phase": "1A", "branch": "feat/grpc-node-command", "classification": "standard", "parallelizableWith": ["1B.1 (other worktree)"], "status": "completed", "blockedBy": [5], "commit": "64774652", "review": "spec ✅ + code ✅"},
{"id": 7, "subject": "Task 1A.2: MeshGrpcCommandDialer + third branch in CentralCommunicationActor (+ security register items 1-2)", "phase": "1A", "branch": "feat/grpc-node-command", "classification": "high-risk", "parallelizableWith": ["1B.2 (other worktree)"], "status": "in_progress", "blockedBy": [6]},
{"id": 8, "subject": "Task 1A.3: 1A live gate (central pair flipped, rig-only, reverted)", "phase": "1A", "branch": "feat/grpc-node-command", "classification": "small", "parallelizableWith": [], "status": "pending", "blockedBy": [7]},
{"id": 9, "subject": "Task 1B.1: CentralCommandGrpcService + admin validator rule", "phase": "1B", "branch": "feat/grpc-central-ack", "classification": "standard", "parallelizableWith": ["1A.1 (other worktree)"], "status": "pending", "blockedBy": [5]},
{"id": 10, "subject": "Task 1B.2: GrpcApplyAckSender + node-side wiring", "phase": "1B", "branch": "feat/grpc-central-ack", "classification": "high-risk", "parallelizableWith": ["1A.2 (other worktree)"], "status": "pending", "blockedBy": [9]},
{"id": 11, "subject": "Task 1B.3: 1B live gate (site-a flipped, rig-only, reverted; rebase on 1A first)", "phase": "1B", "branch": "feat/grpc-central-ack", "classification": "small", "parallelizableWith": [], "status": "pending", "blockedBy": [8, 10]},
{"id": 12, "subject": "Phase 2: baseline reproduction + full-rig cutover + soak + kill drills", "phase": "2", "branch": "(rig commit)", "classification": "high-risk", "parallelizableWith": [], "status": "pending", "blockedBy": [8, 11]},
{"id": 13, "subject": "Task 3.1: delete ClusterClient (code + HOCON + config + tests, ordered)", "phase": "3", "branch": "feat/grpc-phase3-deletion", "classification": "high-risk", "parallelizableWith": [], "status": "pending", "blockedBy": [12]},
{"id": 14, "subject": "Task 3.2: docs sweep (Configuration/Redundancy/Telemetry/CLAUDE.md + scadaproj index)", "phase": "3", "branch": "feat/grpc-phase3-deletion", "classification": "small", "parallelizableWith": [], "status": "pending", "blockedBy": [13]},
{"id": 15, "subject": "Phase 4: terminal live gate on post-deletion image (8 checks, provenance-guarded)", "phase": "4", "branch": "(master)", "classification": "high-risk", "parallelizableWith": [], "status": "pending", "blockedBy": [14]}
{
"id": 0,
"subject": "Task 0.1: mesh_transport.proto + contract lock",
"phase": "0",
"branch": "feat/grpc-phase0-contract-auth",
"classification": "standard",
"parallelizableWith": [
"0.3 (batch, same agent)"
],
"status": "completed",
"commit": "c0ccfa27",
"review": "spec ✅ + code ✅"
},
{
"id": 1,
"subject": "Task 0.2: DTO mapper + round-trip goldens",
"phase": "0",
"branch": "feat/grpc-phase0-contract-auth",
"classification": "high-risk",
"parallelizableWith": [],
"status": "completed",
"blockedBy": [
0
],
"commit": "545413a8",
"review": "spec ✅ + code ✅ (serial)"
},
{
"id": 2,
"subject": "Task 0.3: MeshTransportOptions + validators + SplitTopology relaxation",
"phase": "0",
"branch": "feat/grpc-phase0-contract-auth",
"classification": "standard",
"parallelizableWith": [
"0.1 (batch, same agent)"
],
"status": "completed",
"commit": "aa21f61f",
"review": "spec ✅ + code ✅"
},
{
"id": 3,
"subject": "Task 0.4: node-side MeshCommandAuthInterceptor",
"phase": "0",
"branch": "feat/grpc-phase0-contract-auth",
"classification": "high-risk",
"parallelizableWith": [
"0.5 (batch, same agent)"
],
"status": "completed",
"blockedBy": [
2
],
"commit": "fbda015c + fixes 24e55d33/a4631410",
"review": "spec ✅ + code ✅ (opus, fix loop closed)"
},
{
"id": 4,
"subject": "Task 0.5: IMeshPskProvider + CentralCommandAuthInterceptor",
"phase": "0",
"branch": "feat/grpc-phase0-contract-auth",
"classification": "high-risk",
"parallelizableWith": [
"0.4 (batch, same agent)"
],
"status": "completed",
"blockedBy": [
2
],
"commit": "68a416e4 + fixes 24e55d33/a4631410",
"review": "spec ✅ + code ✅ (opus, fix loop closed); 4 items gated on Phase 1A — see tracking §Phase 1A security register"
},
{
"id": 5,
"subject": "Task 0.6: rig config + FleetStatus issue + docs stub + Phase 0 gate",
"phase": "0",
"branch": "feat/grpc-phase0-contract-auth",
"classification": "small",
"parallelizableWith": [],
"status": "completed",
"blockedBy": [
0,
1,
2,
3,
4
],
"commit": "23555ed0",
"review": "code ✅",
"gate": "PASS 4/4; issues #542 #543 filed"
},
{
"id": 6,
"subject": "Task 1A.1: NodeCommandGrpcService on the telemetry listener",
"phase": "1A",
"branch": "feat/grpc-node-command",
"classification": "standard",
"parallelizableWith": [
"1B.1 (other worktree)"
],
"status": "completed",
"blockedBy": [
5
],
"commit": "64774652",
"review": "spec ✅ + code ✅"
},
{
"id": 7,
"subject": "Task 1A.2: MeshGrpcCommandDialer + third branch in CentralCommunicationActor (+ security register items 1-2)",
"phase": "1A",
"branch": "feat/grpc-node-command",
"classification": "high-risk",
"parallelizableWith": [
"1B.2 (other worktree)"
],
"status": "completed",
"blockedBy": [
6
],
"commit": "14d21496+6a862011 + fixes 48dc2120/86633f14",
"review": "spec ✅ + code ✅ (opus; I-2 fixed via gate-follows-endpoint decision)"
},
{
"id": 8,
"subject": "Task 1A.3: 1A live gate (central pair flipped, rig-only, reverted)",
"phase": "1A",
"branch": "feat/grpc-node-command",
"classification": "small",
"parallelizableWith": [],
"status": "in_progress",
"blockedBy": [
7
]
},
{
"id": 9,
"subject": "Task 1B.1: CentralCommandGrpcService + admin validator rule",
"phase": "1B",
"branch": "feat/grpc-central-ack",
"classification": "standard",
"parallelizableWith": [
"1A.1 (other worktree)"
],
"status": "pending",
"blockedBy": [
5
]
},
{
"id": 10,
"subject": "Task 1B.2: GrpcApplyAckSender + node-side wiring",
"phase": "1B",
"branch": "feat/grpc-central-ack",
"classification": "high-risk",
"parallelizableWith": [
"1A.2 (other worktree)"
],
"status": "pending",
"blockedBy": [
9
]
},
{
"id": 11,
"subject": "Task 1B.3: 1B live gate (site-a flipped, rig-only, reverted; rebase on 1A first)",
"phase": "1B",
"branch": "feat/grpc-central-ack",
"classification": "small",
"parallelizableWith": [],
"status": "pending",
"blockedBy": [
8,
10
]
},
{
"id": 12,
"subject": "Phase 2: baseline reproduction + full-rig cutover + soak + kill drills",
"phase": "2",
"branch": "(rig commit)",
"classification": "high-risk",
"parallelizableWith": [],
"status": "pending",
"blockedBy": [
8,
11
]
},
{
"id": 13,
"subject": "Task 3.1: delete ClusterClient (code + HOCON + config + tests, ordered)",
"phase": "3",
"branch": "feat/grpc-phase3-deletion",
"classification": "high-risk",
"parallelizableWith": [],
"status": "pending",
"blockedBy": [
12
]
},
{
"id": 14,
"subject": "Task 3.2: docs sweep (Configuration/Redundancy/Telemetry/CLAUDE.md + scadaproj index)",
"phase": "3",
"branch": "feat/grpc-phase3-deletion",
"classification": "small",
"parallelizableWith": [],
"status": "pending",
"blockedBy": [
13
]
},
{
"id": 15,
"subject": "Phase 4: terminal live gate on post-deletion image (8 checks, provenance-guarded)",
"phase": "4",
"branch": "(master)",
"classification": "high-risk",
"parallelizableWith": [],
"status": "pending",
"blockedBy": [
14
]
}
],
"lastUpdated": "2026-08-06"
}
}