docs(docker-dev): first-POST-404 caught — Traefik entrypoint-before-dynamic-config window + no-body POST binding verified

Live gate on the local OrbStack docker-dev rig (branch tip images, fresh
down -v + up -d --build).

404: did NOT reproduce through deploy.sh on a fresh boot — the readiness
gate passed on its first poll, and compose's depends_on: service_healthy
serialization means up -d returns ~30 s after Traefik starts, so the window
is already closed. Reproduced it directly instead, by rapid-polling :9200
across a traefik container restart: connection-refused -> 404 for ~65-160 ms
-> 200. Body is exactly Go's "404 page not found" (19 bytes) and the access
log records it with an EMPTY RouterName/ServiceName at 0ms, so no router
matched and no backend was dialled. Mechanism: the entrypoint binds before
the dynamic file-provider config is loaded. The competing candidate (health-
checked service with no healthy server) is excluded — that is a matched
router answering 503, and there are zero 503s in the log. Caveat recorded:
the earliest 404s predate the accesslog handler itself and are not logged,
so an empty access log does not prove absence.

No-body POST: verified, not 400. The minimal-API DeployRequest? binds both
a missing body and a Content-Length: 0 body as null; the handler falls back
to createdBy = "rest-api". A bare ./docker-dev/deploy.sh answered 202
Accepted and the Deployment row carried CreatedBy = rest-api.

deploy.sh's header NOTE goes from "unverified" to the observed truth.
This commit is contained in:
Joseph Doherty
2026-08-18 08:00:47 -04:00
parent 326ca33800
commit 3f04a4d4e5
2 changed files with 55 additions and 16 deletions
+35 -9
View File
@@ -95,7 +95,14 @@ issues the single, un-retried `POST http://localhost:9200/api/deployments` with
`X-Api-Key: docker-dev-deploy-key`. Override the URL / key / timeout via `OTOPCUA_ADMIN_URL`,
`OTOPCUA_DEPLOY_API_KEY`, `OTOPCUA_READY_TIMEOUT_SECONDS`.
### First-POST 404
The body is optional and the no-body form is fully supported (verified on this rig 2026-08-18):
the minimal-API `DeployRequest?` parameter binds a missing body *and* a `Content-Length: 0` body
as **null** — it does not 400 — and the handler defaults the audit field to
`CreatedBy = "rest-api"`. A bare `./docker-dev/deploy.sh` answers `202 Accepted` with the new
`deploymentId` + `revisionHash`, `200` when there is nothing to deploy, and `409` while another
deployment is still in flight.
### First-POST 404 — mechanism CONFIRMED (2026-08-18)
The first headless POST after `up -d` was seen answering **404** ~40 s into boot (2026-08-17 live
gate), clearing on retry. Neither end can produce it: the AdminUI maps every route synchronously
@@ -104,14 +111,33 @@ failure modes are 503 / 401 / 422 / 409 / 500 — it has no 404 path; and Traefi
router whose servers are all down with **503 "no available server"** (the checked-in
`traefik-dynamic.yml` router is `PathPrefix(/)` → `otopcua-admin` with a `/health/active`
healthCheck). Traefik's `api@internal` router (`PathPrefix(/api)`) is bound to the `:8080`
entrypoint (host 8089), not `web`/`:9200`, so it does not shadow `/api/deployments` either. So a
404 from `:9200` is Go's default "404 page not found" — **no router matched at that instant**;
the mechanism is unconfirmed. The `--accesslog=true --accesslog.fields.defaultmode=keep` flags on
the `traefik` service exist to diagnose the next occurrence: `docker compose -f
docker-dev/docker-compose.yml logs traefik` and look for the request with an empty `RouterName`.
The readiness gate in `deploy.sh` avoids the race (a 200 through `:9200` proves both a matched
router and a live backend), so a 404 *after* the gate is new evidence and the script fails loudly
saying so.
entrypoint (host 8089), not `web`/`:9200`, so it does not shadow `/api/deployments` either.
**The mechanism is the window between Traefik binding the `web` entrypoint and loading the
dynamic file-provider config** (`--providers.file.filename=/etc/traefik/dynamic.yml`). The
entrypoint accepts connections *before* the router table exists, and until it does, every request
gets Go's default `http.NotFound` handler. Reproduced 2026-08-18 by rapid-polling `:9200` across a
`docker restart otopcua-dev-traefik-1`: the port goes from connection-refused straight to **404 for
~65–160 ms**, then 200. The response body is exactly `404 page not found` (19 bytes), and the
access log — with the `--accesslog=true --accesslog.fields.defaultmode=keep` flags this service
carries for precisely this purpose — records it as:
```
192.168.97.1 - - [18/Aug/2026:11:58:26 +0000] "GET /health/ready HTTP/1.1" 404 19 "-" "-" 1 "-" "-" 0ms
```
**Empty `RouterName`, empty `ServiceName`, `0ms`** — nothing matched and no backend was dialled.
The competing candidate is **excluded**: a healthCheck'd service with no healthy server yet would
be a *matched* router (`otopcua-admin@file`) answering **503**, and zero 503s appear anywhere in
the log. Note the window has two sub-phases — the earliest 404s are not access-logged at all
(the accesslog handler is itself part of the not-yet-built dynamic chain), later ones are; both are
the same client-visible 404, so **absence from the access log does not mean absence of the 404**.
The readiness gate in `deploy.sh` closes the race (it loops on any non-200, and a 200 through
`:9200` proves both a matched router and a live backend), so a 404 *after* the gate is new evidence
and the script fails loudly saying so. A plain `docker compose up -d` also hides the window on this
rig without the gate helping: `up -d` blocks on the `depends_on: service_healthy` conditions and
returns ~30 s *after* Traefik starts, by which time the router table is long loaded.
## Tear down
+20 -7
View File
@@ -8,23 +8,36 @@
# end of the request: the AdminUI maps every route synchronously before RunAsync and
# the deploy endpoint has no 404 path (503/401/422/409/500 only), and Traefik answers
# a matched-router-with-all-servers-down as 503 "no available server". So the 404 means
# no Traefik router matched at that instant. Polling /health/ready through :9200 until
# it answers 200 proves both halves are in place — the router table carries the
# otopcua-admin router AND a backend is up — before the single, un-retried POST.
# no Traefik router matched at that instant.
#
# CONFIRMED 2026-08-18: the window is between Traefik binding the `web` entrypoint and
# loading its dynamic file-provider config — the port accepts connections before the
# router table exists, so requests get Go's default `404 page not found` (19 bytes),
# access-logged with an EMPTY RouterName/ServiceName and 0ms. Measured at ~65-160 ms
# across a `docker restart` of the traefik container. See docker-dev/README.md
# §"First-POST 404". Polling /health/ready through :9200 until it answers 200 proves
# both halves are in place — the router table carries the otopcua-admin router AND a
# backend is up — before the single, un-retried POST.
#
# A 404 AFTER the gate is therefore new evidence, not the known boot race: it should be
# investigated against the Traefik access log (`--accesslog=true` on the traefik service;
# `docker compose -f docker-dev/docker-compose.yml logs traefik`), which records the
# RouterName (empty on an unmatched request) for the failing call.
# RouterName (empty on an unmatched request) for the failing call. Caveat: the earliest
# 404s of the window predate the accesslog handler itself and are NOT logged at all, so
# an empty access log does not prove the 404 did not happen.
#
# Usage:
# ./deploy.sh # no body (matches the bare README snippet)
# ./deploy.sh artifact.json # send that file as the JSON request body
# cat artifact.json | ./deploy.sh
#
# NOTE (unverified against a live rig): the no-body invocation sends
# `Content-Type: application/json` with an empty body. Whether the minimal-API
# `DeployRequest?` parameter binds that as null or answers 400 has not been confirmed.
# NOTE (verified on the docker-dev rig 2026-08-18): the no-body invocation sends
# `Content-Type: application/json` with no body (or `Content-Length: 0` when stdin is a
# redirected-but-empty non-tty, which takes the `--data-binary @-` branch below). The
# minimal-API `DeployRequest?` parameter binds BOTH shapes as null — it does NOT 400 —
# and the handler falls back to `createdBy = "rest-api"` (DeployApiEndpoints.cs). Confirmed
# end to end: a bare `./deploy.sh` returned `HTTP 202 {"outcome":"Accepted",...}` and the
# resulting Deployment row carried `CreatedBy = rest-api`.
#
set -euo pipefail