docs(plans): Phase 2 baseline REPRODUCED — cross-mesh :4053 open unauthenticated; site-b drop suppressed into sampled deadLetters INF, deadline sole witness
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
This commit is contained in:
@@ -40,8 +40,61 @@ no per-node dispatch error surfaces anywhere; site-b's `NodeDeploymentState` row
|
||||
`Applying` until the 2-minute deadline marks the deployment `TimedOut`/partially failed. The
|
||||
deadline is the only witness. Restart the pair.
|
||||
|
||||
- [ ] Baseline (a) recorded with evidence — **PENDING**
|
||||
- [ ] Baseline (b) recorded with evidence — **PENDING**
|
||||
- [x] Baseline (a) recorded with evidence — **REPRODUCED** 2026-08-06, rig on `master @ 46be4c8d`
|
||||
(post-1B image, all-ClusterClient default, no mode override; `MeshTransport__Mode=ClusterClient`
|
||||
confirmed via `printenv` in the site container)
|
||||
- [x] Baseline (b) recorded with evidence — **REPRODUCED** same run
|
||||
|
||||
### Baseline (a) evidence — unauthenticated Akka remoting crosses the boundary
|
||||
|
||||
```
|
||||
$ docker exec otopcua-dev-site-a-1-1 bash -c 'exec 3<>/dev/tcp/central-1/4053 && echo OPEN'
|
||||
OPEN # no credential presented on this path
|
||||
$ docker logs otopcua-dev-central-1-1 | grep -i "clusterclient\|receptionist" | tail -5
|
||||
[13:12:51 INF] Connected to [akka://otopcua/system/receptionist]
|
||||
[13:12:51 INF] Mesh ClusterClient for cluster SITE-B created with 2 contact point(s)
|
||||
[13:12:53 INF] Connected to [akka://otopcua/system/receptionist]
|
||||
[13:12:54 INF] Connected to [akka.tcp://otopcua@site-a-1:4053/system/receptionist]
|
||||
[13:12:57 INF] Connected to [akka.tcp://otopcua@site-b-1:4053/system/receptionist]
|
||||
```
|
||||
|
||||
### Baseline (b) evidence — silent drop on an unreachable cluster
|
||||
|
||||
Site-b pair stopped 13:53:5x UTC; dispatch accepted 13:53:54
|
||||
(`POST /api/deployments` → `{"outcome":"Accepted","deploymentId":"c6cd0798-…"}`, HTTP 202).
|
||||
|
||||
```
|
||||
# 20 s after dispatch — 4/6 applied within ~15 ms, site-b rows parked, deployment open:
|
||||
NodeDeploymentState: central-1/-2, site-a-1/-2 → Status 1 @ 13:53:54.9xx
|
||||
site-b-1/-2 → Status 0, AppliedAtUtc NULL
|
||||
Deployment c6cd0798 → Status 1 (in progress)
|
||||
|
||||
# The site-b fan leg is not merely unlogged — it is actively suppressed. The SITE-B
|
||||
# ClusterClient's SendToAll lands in deadLetters at INF, under Akka's sampled generic
|
||||
# dead-letter counter, naming no node and no failure:
|
||||
[13:53:54 INF] Message [DispatchDeployment] wrapped in [$SendToAll] … to [akka://otopcua/deadLetters]
|
||||
was not delivered. [467] dead letters encountered, of which 456 were not logged. …
|
||||
Message content: DispatchDeployment { DeploymentId = c6cd0798…, … }
|
||||
|
||||
# Dispatch-to-deadline window (13:53:50–13:55:53): ambient Akka association-gating noise for
|
||||
# the downed pair exists (transport chatter, fires with or without a dispatch), but ZERO lines
|
||||
# correlate site-b with the deployment/dispatch/command:
|
||||
$ docker logs --since …09:53:50 --until …09:55:53 otopcua-dev-central-1-1 \
|
||||
| grep -i "site-b" | grep -icE "deploy|dispatch|command|c6cd0798"
|
||||
0
|
||||
|
||||
# The deadline is the only witness, exactly 2 minutes later:
|
||||
[13:55:54 WRN] Deployment c6cd0798… timed out after 00:02:00 (4/6 acks landed). No ack from:
|
||||
site-b-1:4053, site-b-2:4053. Each is an enabled, non-maintenance ClusterNode
|
||||
row — start the node, or set ClusterNode.MaintenanceMode = 1 …
|
||||
Deployment c6cd0798 → Status 4; site-b rows stay Status 0/NULL (a timed-out deployment is not
|
||||
retro-acked; boot replay serves the latest artifact without resurrecting the dead record).
|
||||
```
|
||||
|
||||
Pair restarted after the drill; site-b-1 back healthy. Note for later log reads on this host:
|
||||
`docker logs --since/--until` parses bare timestamps as HOST-LOCAL time (EDT here, UTC−4) while
|
||||
the app writes UTC — a UTC-shaped `--since` silently returns nothing (it points 4 h into the
|
||||
future). Convert to local first; earlier in this run that mistake made a window look empty.
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user