From 1bb3a937ae36e8e3c6f733ce0a04fa0cc94286fd Mon Sep 17 00:00:00 2001
From: Joseph Doherty <dohejw01@gmail.com>
Date: Thu, 6 Aug 2026 09:58:09 -0400
Subject: [PATCH] =?UTF-8?q?docs(plans):=20Phase=202=20baseline=20REPRODUCE?=
 =?UTF-8?q?D=20=E2=80=94=20cross-mesh=20:4053=20open=20unauthenticated;=20?=
 =?UTF-8?q?site-b=20drop=20suppressed=20into=20sampled=20deadLetters=20INF?=
 =?UTF-8?q?,=20deadline=20sole=20witness?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
---
 ...6-08-06-clusterclient-to-grpc-live-gate.md | 57 ++++++++++++++++++-
 1 file changed, 55 insertions(+), 2 deletions(-)

diff --git a/docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md b/docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md
index 64f68181..784a13ff 100644
--- a/docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md
+++ b/docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md
@@ -40,8 +40,61 @@ no per-node dispatch error surfaces anywhere; site-b's `NodeDeploymentState` row
 `Applying` until the 2-minute deadline marks the deployment `TimedOut`/partially failed. The
 deadline is the only witness. Restart the pair.
 
-- [ ] Baseline (a) recorded with evidence — **PENDING**
-- [ ] Baseline (b) recorded with evidence — **PENDING**
+- [x] Baseline (a) recorded with evidence — **REPRODUCED** 2026-08-06, rig on `master @ 46be4c8d`
+  (post-1B image, all-ClusterClient default, no mode override; `MeshTransport__Mode=ClusterClient`
+  confirmed via `printenv` in the site container)
+- [x] Baseline (b) recorded with evidence — **REPRODUCED** same run
+
+### Baseline (a) evidence — unauthenticated Akka remoting crosses the boundary
+
+```
+$ docker exec otopcua-dev-site-a-1-1 bash -c 'exec 3<>/dev/tcp/central-1/4053 && echo OPEN'
+OPEN                                          # no credential presented on this path
+$ docker logs otopcua-dev-central-1-1 | grep -i "clusterclient\|receptionist" | tail -5
+[13:12:51 INF] Connected to [akka://otopcua/system/receptionist]
+[13:12:51 INF] Mesh ClusterClient for cluster SITE-B created with 2 contact point(s)
+[13:12:53 INF] Connected to [akka://otopcua/system/receptionist]
+[13:12:54 INF] Connected to [akka.tcp://otopcua@site-a-1:4053/system/receptionist]
+[13:12:57 INF] Connected to [akka.tcp://otopcua@site-b-1:4053/system/receptionist]
+```
+
+### Baseline (b) evidence — silent drop on an unreachable cluster
+
+Site-b pair stopped 13:53:5x UTC; dispatch accepted 13:53:54
+(`POST /api/deployments` → `{"outcome":"Accepted","deploymentId":"c6cd0798-…"}`, HTTP 202).
+
+```
+# 20 s after dispatch — 4/6 applied within ~15 ms, site-b rows parked, deployment open:
+NodeDeploymentState: central-1/-2, site-a-1/-2 → Status 1 @ 13:53:54.9xx
+                     site-b-1/-2               → Status 0, AppliedAtUtc NULL
+Deployment c6cd0798 → Status 1 (in progress)
+
+# The site-b fan leg is not merely unlogged — it is actively suppressed. The SITE-B
+# ClusterClient's SendToAll lands in deadLetters at INF, under Akka's sampled generic
+# dead-letter counter, naming no node and no failure:
+[13:53:54 INF] Message [DispatchDeployment] wrapped in [$SendToAll] … to [akka://otopcua/deadLetters]
+               was not delivered. [467] dead letters encountered, of which 456 were not logged. …
+               Message content: DispatchDeployment { DeploymentId = c6cd0798…, … }
+
+# Dispatch-to-deadline window (13:53:50–13:55:53): ambient Akka association-gating noise for
+# the downed pair exists (transport chatter, fires with or without a dispatch), but ZERO lines
+# correlate site-b with the deployment/dispatch/command:
+$ docker logs --since …09:53:50 --until …09:55:53 otopcua-dev-central-1-1 \
+    | grep -i "site-b" | grep -icE "deploy|dispatch|command|c6cd0798"
+0
+
+# The deadline is the only witness, exactly 2 minutes later:
+[13:55:54 WRN] Deployment c6cd0798… timed out after 00:02:00 (4/6 acks landed). No ack from:
+               site-b-1:4053, site-b-2:4053. Each is an enabled, non-maintenance ClusterNode
+               row — start the node, or set ClusterNode.MaintenanceMode = 1 …
+Deployment c6cd0798 → Status 4; site-b rows stay Status 0/NULL (a timed-out deployment is not
+retro-acked; boot replay serves the latest artifact without resurrecting the dead record).
+```
+
+Pair restarted after the drill; site-b-1 back healthy. Note for later log reads on this host:
+`docker logs --since/--until` parses bare timestamps as HOST-LOCAL time (EDT here, UTC−4) while
+the app writes UTC — a UTC-shaped `--since` silently returns nothing (it points 4 h into the
+future). Convert to local first; earlier in this run that mistake made a window look empty.
 
 ---