# 20 ways a deploy lies to you

Each entry starts with what you see, then the cause, the fix and the prevention. Every one of them appeared in production first.

Twenty ways a deployment looks healthy while being broken. Each entry starts with the symptom you can search for, then the cause, the fix and the prevention.

The Docker Compose backup setup behind several of the failures below is documented in [Easy database backups with Docker Compose](/posts/easy-database-backups-with-docker-compose).

## Traefik dashboard reachable from the internet

**Symptom.** `curl -H "Host: traefik.internal.example.com" https://203.0.113.10/api/overview` returns 200 JSON from a laptop on a random network.

| Cause | Fix | Prevention |
| --- | --- | --- |
| dashboard router enabled with `ipwhitelist.sourcerange=0.0.0.0/0`, plus an origin reachable directly. The staging hostname was never in DNS, so nobody noticed it was reachable by Host header. | replace the allowlist with tailnet plus loopback, add basic auth, remove the public cert resolver for that router, verify 401 from tailnet and a drop from the internet. | exposure scan in the go-live checklist, dashboard middleware in the kit, quarterly external scan. |

## Traefik /metrics exposed on the public entrypoint

**Symptom.** `https://app.example.com/metrics` returns Prometheus text instead of the app's 404.

| Cause | Fix | Prevention |
| --- | --- | --- |
| `metrics.prometheus.entryPoint: websecure` in the static config. Traefik auto-creates a `prometheus@internal` router and publishes it on the public entrypoint. Nobody wrote a route. | remove the entryPoint key. Metrics stay on the internal `traefik` entrypoint, which is not published. | check the router list for internal routers on public entrypoints after every Traefik upgrade. |

## Origin serves TRAEFIK DEFAULT CERT

**Symptom.** origin serves `CN=TRAEFIK DEFAULT CERT`; the public site still works because Cloudflare is set to Full, not Full strict.

| Cause | Fix | Prevention |
| --- | --- | --- |
| the Cloudflare token was mounted as a Docker secret, but lego reads `CLOUDFLARE_DNS_API_TOKEN` from the environment. `acme.json` never had a certificate; every start logged "credentials information are missing". | pass the token as env, restart, confirm `acme.json` contents and the issuer, then set Cloudflare to Full strict. | the proof table checks the issuer, and the ACME check is part of go-live. |

## ACME fails with a Cloudflare 403 Authentication error

**Symptom.** ACME fails with `403 10000 Authentication error` when creating the TXT record.

| Cause | Fix | Prevention |
| --- | --- | --- |
| the Cloudflare token can read zones but lacks `Zone.DNS.Edit` on the target zone. Reading zones successfully makes it look healthy. | create a token with `Zone.Zone.Read` plus `Zone.DNS.Edit` on the specific zone, store it only as env, restart. | document required scopes next to the token, never reuse a read-only token for ACME. |

## Backup sidecar runs but produces no dump

**Symptom.** backup sidecar is "up", but the last dump is months old; logs show a connection attempt every 30 seconds.

| Cause | Fix | Prevention |
| --- | --- | --- |
| a compose project rename changed the database container name. The sidecar's `DB01_HOST` still pointed at the old name, which no longer resolved. Restart policy kept it alive, so nothing alerted. | point the sidecar at the compose service name, add a healthcheck that fails when the last successful dump is older than the interval, and alert on it. | staleness alert on the backup directory, restore drill monthly, do not rely on container "Up" as a backup health signal. |

## Backup container exits and the backups folder stays empty

**Symptom.** the backup container starts and exits; the host `backups/` directory stays empty.

| Cause | Fix | Prevention |
| --- | --- | --- |
| the image tag is deprecated and the volume mount path no longer matches the image's expectation. | switch to the maintained fork (`nfrastack/container-db-backup`), verify the mount path, run a manual backup, check the file. | pin image versions, test a manual run on install, alert on empty or stale output. |

## Offsite copy is months behind while local backups look healthy

**Symptom.** local backups look healthy, but the offsite copy is months behind.

| Cause | Fix | Prevention |
| --- | --- | --- |
| the offsite peer (send-only Syncthing or an rclone target) went offline and nothing alerted. The local job kept succeeding. | alert on the remote freshness, not the local job. Add a heartbeat from the offsite target. | 3-2-1 with a check on the third copy, not just the first two. |

## Git host has no database dump

**Symptom.** 20+ GB of repositories on the git host, no database dump, no forge dump. Mirrors exist, but they are copies of the same data.

| Cause | Fix | Prevention |
| --- | --- | --- |
| the git host "is the source of truth", so nobody backed it up. | scheduled forge dump plus database dump, rotated offsite, restore drill into a throwaway instance. | every stateful service on the box appears in the backup matrix with an owner and a restore command. |

## Rotated secret never reached the host .env

**Symptom.** a background job starts failing authentication days after a secret rotation. The app's config shows the new value; the host env has the old one.

| Cause | Fix | Prevention |
| --- | --- | --- |
| the secret manager was updated but the host `.env` was not, and nothing compares the two. | update the host env, recreate the containers, verify with a real API call. | one documented source of truth per app, a preflight that validates critical third-party credentials, and a rotation checklist. |

## Two pipelines deploy the same app

**Symptom.** a deploy happens that nobody triggered from the new pipeline. A feature is briefly live, then disappears.

| Cause | Fix | Prevention |
| --- | --- | --- |
| both a legacy GitHub Actions workflow and the new CI pipeline deploy the same stack. | disable or delete the legacy pipeline, keep one deploy path, document it in `DEPLOY.md`. | CI inventory per repo, tag naming convention, deploy key restricted to the active pipeline only. |

## CI is green but the running image did not change

**Symptom.** the pipeline says success; the running image digest is unchanged.

| Cause | Fix | Prevention |
| --- | --- | --- |
| the deploy step ran `docker compose up` with a compose file named `compose.yml` while the stack reads `compose.yaml`, or the registry hostname did not resolve inside the job and the deploy step exited early with a masked error. | assert the digest before and after, ship the exact filename, fail on empty tag variables. | digest assertion in `deploy.sh`, no-op warning, external smoke test after deploy. |

## /up returns 200 while pages return 500

**Symptom.** `/up` returns 200 while a critical endpoint returns 500 and users complain.

| Cause | Fix | Prevention |
| --- | --- | --- |
| the health route returns a static response and touches no dependencies. | make `/up` check the database, cache, and storage; add a post-deploy exception check for one critical user path. | dependency-aware health route as part of the app contract, exception gate after deploy. |

## Rollback has no immutable tag to pin

**Symptom.** rollback is requested and there is nothing to roll back to, because `latest` moved.

| Cause | Fix | Prevention |
| --- | --- | --- |
| the pipeline only publishes mutable tags. | publish immutable `vX.Y.Z-web` and `vX.Y.Z-worker` via manifest re-tag, keep them for a year, document `--rollback TAG`. | rollback drill quarterly, immutable tag step in the pipeline kit. |

## Whitelist entry silently disables CrowdSec

**Symptom.** a scanner walks past bot protection on every host, or a crawler gets captcha pages in monitoring.

| Cause | Fix | Prevention |
| --- | --- | --- |
| during an incident someone widened the whitelist (an IP, a user agent, or a CIDR) and never narrowed it. | diff the whitelist against the documented baseline, remove the extra entry, reload CrowdSec. | whitelist canary that proves it still discriminates, incident checklist line item, quarterly review. |

## Web container crash-loops and Caddy answers 404

**Symptom.** the web container crash-loops, Caddy answers every request with its own plain-text 404, and the PHP test suite is green.

| Cause | Fix | Prevention |
| --- | --- | --- |
| the bundled FrankenPHP is below the floor required by Octane. Octane tries to replace its own binary, fails as a non-root user, and exits. | bump the runtime image or hold Octane below the release that raised the floor; add the build-time floor guard. | the floor check runs at image build, not in production. |

## One slow request stalls the whole site

**Symptom.** latency spikes under a single slow endpoint, even though the CPU limit looks generous.

| Cause | Fix | Prevention |
| --- | --- | --- |
| `octane:frankenphp --workers` had no effect because the Caddyfile had no `frankenphp { worker { } }` block. The PHP thread pool collapsed to two threads under the CPU limit. | add the worker block with `{$CADDY_SERVER_WORKER_DIRECTIVE:num 6}`, raise the CPU limit to 2, keep `--max-requests`. | load test one slow route after the first deploy, document the CPU/worker relationship. |

## Disk fills and writes start failing

**Symptom.** writes fail, MySQL goes read-only, containers restart, logs disappear.

| Cause | Fix | Prevention |
| --- | --- | --- |
| docker json-file logs without rotation, application logs growing forever, backups on the same disk, old images never pruned. | free space, add log rotation, prune images, move backups or rotate harder. | disk alert at 70 percent, log rotation in every compose service, image prune in maintenance loop. |

## Deploy fails halfway through migrations

**Symptom.** deploy fails halfway through migrations, or tests pass on SQLite while production MySQL throws.

| Cause | Fix | Prevention |
| --- | --- | --- |
| migrations tested against a different engine, or non-idempotent DDL that partially applied. | restore the pre-deploy dump, fix the migration to be idempotent, rerun against production engine. | CI test gate on the production engine, pre-migration dump, forward-only migrations with `hasColumn` guards. |

## Docs describe a stack that no longer exists

**Symptom.** the next agent follows the docs and deploys the wrong thing, or asks questions that were answered six months ago.

| Cause | Fix | Prevention |
| --- | --- | --- |
| the stack changed, the docs did not. Some apps moved to Doppler, some stayed on env files. | rewrite `DEPLOY.md` from the actual running configuration, not from memory. | quarterly drift review, AAR after every incident, docs as part of the definition of done. |
