# Deploy by pulling verified images

A deploy either verifies success or fails loudly. The script pulls a prebuilt image, gates on health, and has a rollback path.

By the end of this page you have one deploy script that works for every app, fails loudly, restores the site when a step fails, refuses to deploy a half-configured stack, and gives you a one-line rollback.

Everything before this page produced an image. This chapter is about the twenty seconds where that image becomes the thing users hit, which is the twenty seconds most incidents live in.

## The server layout

One directory per app, owned by the deploy user. Nothing else on the box knows about it.

```
/srv/app/
├── compose.yaml               # shipped by CI on every deploy
├── deploy.sh                  # shipped by CI on every deploy
├── .env                       # operator-managed, mode 600, never in git
├── .registry.env              # registry credentials, mode 600, written by CI
├── .storage/                  # bind-mounted app storage (logs, app, framework)
│   ├── logs/
│   ├── app/
│   └── framework/
├── .postgres-db/              # bind-mounted database data
├── backups/                   # database dumps (or a volume)
└── compose.rollback.yaml      # generated only while a rollback is pinned
```

The app's git repository is not needed on the host at all. CI ships the two files it owns. The host owns config and data.

## Why the deploy is pull-only

The deploy script never runs `docker build`, `composer install`, `npm ci`, or `git pull`. It pulls a prebuilt image, replaces containers, migrates, and verifies.

This is not a style preference. It means:

- the bytes that ran in tests are the bytes that run in production;
- the production host needs no compilers, no Node, and no Composer;
- a release takes seconds, not minutes, so rollback is cheap;
- a failed build cannot take the site down, because it happens somewhere else.

If a deploy needs a rebuild, that is a CI trigger, not a deploy step.

## The sequence

Every script in this guide follows the same contract. Read the order carefully, because the order is the safety.

1. **Validate config before touching anything.** Missing `APP_KEY` or a database password should fail before maintenance mode, not halfway through.
2. **Capture the running image digest.** This makes a no-op deploy visible.
3. **Log in to the registry and pull.** Pull before maintenance mode so a registry outage does not take the site down for nothing.
4. **Prepare storage bind mounts.** The container runs as a non-root user; the host deploy user cannot `chown` to it. Do it from a throwaway root container.
5. **Enable maintenance mode.** Short, honest downtime beats a half-broken rollout on a single box. Skip it only for stateless apps or multi-replica rolling updates.
6. **Recreate the app containers.** `--force-recreate` on the long-running processes, so Octane or the queue worker actually boots the new code.
7. **Run migrations.** `--force`, forward-only, never on rollback.
8. **Run the app's post-deploy hook.** Cache warm, search index, backfill. A file in the repo (`scripts/post-deploy.sh`) keeps this generic.
9. **Leave maintenance mode with retries.** A failed `artisan up` must never be permanent.
10. **Probe `/up` from inside the container, with retries.** Health is a dependency-aware route, not a TCP check.
11. **Warn if the image digest did not change.** A deploy that changed nothing should say so.
12. **Notify and exit non-zero on failure.** A silent failure is how backups stay broken for a year.

The failure invariant is the important part: if anything between step 5 and step 9 fails, a trap takes the site out of maintenance mode and the script exits non-zero, so CI shows red.

## The script

This is the canonical version. Copy it into `scripts/deploy.sh` in every repo, adjust `REQUIRED_VARS` and the health URL, and never fork it per app without a reason.

```bash
#!/usr/bin/env bash
# deploy.sh - pull prebuilt images and deploy this app on its host.
# Runs ON the host, invoked by CI (or by a forced-command wrapper).
set -euo pipefail

APP_DIR="${APP_DIR:-/srv/app}"
COMPOSE_FILE="${COMPOSE_FILE:-$APP_DIR/compose.yaml}"
REGISTRY="${REGISTRY:-${REGISTRY_HOST}}"
IMAGE_REPO="${IMAGE_REPO:-${REGISTRY}/${ORG}/${APP}}"
HEALTH_PATH="${HEALTH_PATH:-/up}"
SERVICES=(web worker scheduler)
NTFY_URL="${NTFY_URL:-}"

cd "$APP_DIR"

# ---------------------------------------------------------------- helpers
red()   { printf '\033[0;31m%s\033[0m\n' "$*" >&2; }
green() { printf '\033[0;32m%s\033[0m\n' "$*"; }
warn()  { printf '\033[1;33mWARNING: %s\033[0m\n' "$*" >&2; }

notify() {
  local title="$1" message="$2" priority="${3:-low}"
  [ -n "$NTFY_URL" ] || return 0
  curl -fsS --max-time 10 -H "Title: $title" -H "Priority: $priority" \
    -d "$message" "$NTFY_URL" >/dev/null 2>&1 || true
}

# ---------------------------------------------------------------- preflight
REQUIRED_VARS=(APP_KEY APP_ENV APP_URL APP_DOMAIN DB_CONNECTION DB_HOST DB_PORT
               DB_DATABASE DB_USERNAME DB_PASSWORD REDIS_HOST REDIS_PORT
               QUEUE_CONNECTION CACHE_STORE)

[ -f "$COMPOSE_FILE" ] || { red "missing $COMPOSE_FILE"; exit 1; }
[ -f .env ] || { red "missing $APP_DIR/.env"; exit 1; }

set -a
# shellcheck disable=SC1091
. ./.env
set +a

missing=()
for var in "${REQUIRED_VARS[@]}"; do
  [ -n "${!var:-}" ] || missing+=("$var")
done
if [ "${#missing[@]}" -gt 0 ]; then
  red "missing or empty required vars in .env:"
  printf '  - %s\n' "${missing[@]}" >&2
  exit 1
fi

if [ "$APP_ENV" != "production" ]; then
  red "APP_ENV must be production, found '$APP_ENV'"
  exit 1
fi
if grep -qE '^APP_KEY=base64:dummy' .env; then
  red "APP_KEY is the build-time dummy. Refusing to deploy."
  exit 1
fi

MODE="deploy"
ROLLBACK_TAG=""
case "${1:-}" in
  --check)    MODE="check" ;;
  --rollback) MODE="rollback"; ROLLBACK_TAG="${2:-}"
              [ -n "$ROLLBACK_TAG" ] || { red "--rollback needs a tag"; exit 2; } ;;
  "")         ;;
  *)          red "unknown argument: $1"; exit 2 ;;
esac

compose() { docker compose -f "$COMPOSE_FILE" "$@"; }

# Digest of the currently running web container, to detect no-op deploys.
current_image() {
  docker inspect --format '{{.Image}}' "$(compose ps -q web 2>/dev/null)" 2>/dev/null || true
}

# ---------------------------------------------------------------- check mode
if [ "$MODE" = "check" ]; then
  green "env OK (${#REQUIRED_VARS[@]} required vars present)"
  compose config -q && green "compose config OK"
  compose config --images | sed 's/^/  /'
  exit 0
fi

# ---------------------------------------------------------------- rollback
if [ "$MODE" = "rollback" ]; then
  green "rolling back to $ROLLBACK_TAG (migrations are NOT run)"
  cat > compose.rollback.yaml <<EOF
services:
  web:
    image: ${IMAGE_REPO}:${ROLLBACK_TAG}-web
  worker:
    image: ${IMAGE_REPO}:${ROLLBACK_TAG}-worker
  scheduler:
    image: ${IMAGE_REPO}:${ROLLBACK_TAG}-worker
EOF
  COMPOSE_FILE="$APP_DIR/compose.yaml"
  docker compose -f "$COMPOSE_FILE" -f compose.rollback.yaml pull
  docker compose -f "$COMPOSE_FILE" -f compose.rollback.yaml up -d --remove-orphans
  for attempt in 1 2 3 4 5 6 7 8 9 10; do
    if compose exec -T web sh -c "curl -fsS -o /dev/null http://127.0.0.1:80${HEALTH_PATH}"; then
      green "/up OK after rollback"
      notify "Rollback complete" "app rolled back to ${ROLLBACK_TAG}" default
      exit 0
    fi
    sleep 3
  done
  red "rollback started but /up never recovered"
  compose logs --tail=60 web >&2 || true
  notify "Rollback failed" "app rollback to ${ROLLBACK_TAG} did not pass /up" high
  exit 1
fi

# ---------------------------------------------------------------- deploy
DEPLOY_START=$(date +%s)
notify "Deploy started" "APP deploy started" low

cleanup() {
  local rc=$?
  if [ "$rc" -ne 0 ]; then
    red "deploy failed (rc=$rc), restoring maintenance mode"
    compose exec -T web php artisan up >/dev/null 2>&1 || true
    compose logs --tail=80 web >&2 || true
    notify "Deploy failed" "deploy failed with rc=$rc; app restored" high
  fi
  exit "$rc"
}
trap cleanup EXIT

# 1. registry login (credentials shipped by CI, mode 600)
if [ -f .registry.env ]; then
  set -a
  # shellcheck disable=SC1091
  . ./.registry.env
  set +a
fi
if [ -n "${FORGEJO_REGISTRY_TOKEN:-}" ]; then
  printf '%s' "$FORGEJO_REGISTRY_TOKEN" |
    docker login "$REGISTRY" -u "${FORGEJO_REGISTRY_USER:?}" --password-stdin
fi

green "[1/9] pulling images"
before="$(current_image)"
compose pull

green "[2/9] preparing storage bind mounts"
for image in "${IMAGE_REPO}:web" "${IMAGE_REPO}:worker"; do
  docker run --rm --user 0 -v "$APP_DIR/.storage:/app/storage" \
    --entrypoint sh "$image" -c \
    'mkdir -p /app/storage/logs /app/storage/app/public /app/storage/framework && chown -R 82:82 /app/storage'
done

green "[3/9] enabling maintenance mode"
compose exec -T web php artisan down --retry=60 2>/dev/null ||
  warn "app not running yet, skipping maintenance mode"

green "[4/9] recreating containers"
compose up -d --remove-orphans --force-recreate "${SERVICES[@]}"

green "[5/9] running migrations"
compose exec -T web php artisan migrate --force

green "[6/9] running post-deploy hook"
if [ -x scripts/post-deploy.sh ]; then
  ./scripts/post-deploy.sh
else
  warn "no scripts/post-deploy.sh, skipping cache warm/backfill"
fi

green "[7/9] leaving maintenance mode"
up_attempt=1
while [ "$up_attempt" -le 6 ]; do
  if compose exec -T web php artisan up; then
    break
  fi
  warn "artisan up failed (attempt ${up_attempt}/6), retrying"
  up_attempt=$((up_attempt + 1))
  sleep 5
done
[ "$up_attempt" -le 6 ] || { red "app may still be in maintenance mode"; exit 1; }

green "[8/9] probing ${HEALTH_PATH}"
status="000"
for attempt in 1 2 3 4 5 6 7 8 9 10; do
  status="$(compose exec -T web sh -c \
    "curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:80${HEALTH_PATH}")" || status="000"
  case "$status" in 2*|3*) break ;; esac
  sleep 3
done
case "$status" in
  2*|3*) green "health gate passed ($status)" ;;
  *) red "health gate failed (last HTTP $status)"; exit 1 ;;
esac

green "[9/9] reporting"
after="$(current_image)"
if [ -n "$before" ] && [ "$before" = "$after" ]; then
  warn "image digest unchanged ($after), this deploy changed nothing"
fi

trap - EXIT
duration=$(( $(date +%s) - DEPLOY_START ))
compose ps
green "deployed in ${duration}s"
notify "Deploy complete" "APP deployed in ${duration}s" default
```

A few details that are easy to miss:

- The digest check matters. If CI shipped the wrong compose filename, or the tag did not move, the digest is identical and the script says so instead of reporting success.
- `artisan up` runs both in the trap and in step 7 on purpose. The trap covers failures between them; the explicit retry covers a slow database. Both are cheap.
- The rollback branch deliberately does not run migrations. Migrations are forward-only in this design. If a release needs a destructive migration, the rollback plan is a database restore, and that is a separate documented operation.
- The storage block runs before maintenance mode and before recreate, so the app never starts against an unwritable framework directory.

## Handling CI invocation

CI ships exactly two files and then calls the script.

```yaml
- name: deploy
  when:
    - event: tag
      ref: 'refs/tags/v*'
  commands:
    - apk add --no-cache openssh-client
    - mkdir -p ~/.ssh && chmod 700 ~/.ssh
    - printf '%s\n' "$DEPLOY_SSH_KEY" > ~/.ssh/id_ed25519
    - chmod 600 ~/.ssh/id_ed25519
    - scp -O -q -i ~/.ssh/id_ed25519 compose.yaml "deploy@${DEPLOY_HOST}:${APP_DIR}/compose.yaml"
    - scp -O -q -i ~/.ssh/id_ed25519 scripts/deploy.sh "deploy@${DEPLOY_HOST}:${APP_DIR}/deploy.sh"
    - scp -O -q -i ~/.ssh/id_ed25519 .registry.env "deploy@${DEPLOY_HOST}:${APP_DIR}/.registry.env"
    - ssh -i ~/.ssh/id_ed25519 "deploy@${DEPLOY_HOST}" "cd ${APP_DIR} && chmod +x deploy.sh && ./deploy.sh"
```

Better: lock the deploy key down with a forced command on the host, so a leaked CI secret can only run the deploy, not open a shell.

```bash
# /home/deploy/.ssh/authorized_keys
command="cd /srv/app && ./deploy.sh",no-agent-forwarding,no-port-forwarding,no-pty ssh-ed25519 AAAA... ci-deploy
```

And on the host side, `.registry.env` must be mode 600. CI writes it, scp preserves 600 only if it was created that way, so do `install -m 600` or `chmod` in the script.

## Zero downtime, honestly

On a single cheap VPS with one database and one web container, true zero downtime is mostly a story people tell. The honest options, in order of cost:

1. **Maintenance mode (this guide).** 10 to 40 seconds of 503 with `Retry-After`. Simple, safe, and fine for a blog, a directory, or an internal tool. Cloudflare can serve a cached version of the page during the window if you enable "cache everything" with a bypass for cookies.
2. **Two replicas behind Traefik and a rolling recreate.** `docker compose up -d --scale web=2` with Traefik health checks, then replace one at a time. Works only if the app is truly stateless and any one replica can serve while the other boots, and it doubles memory on a small box.
3. **Separate release directories and an atomic Traefik switch.** More moving parts, one more thing to get wrong. Worth it only for a revenue-critical service.

Do not claim zero downtime because the container starts fast. The database migration and the maintenance window are where the outage lives.

## Rollback

Two rules make rollback possible.

First, CI pushes two tags per build: a mutable pointer used by normal deploys, and an immutable tag used by rollbacks.

```
${REGISTRY}/${ORG}/${APP}:web              # moves on every release
${REGISTRY}/${ORG}/${APP}:v1.4.2-web       # never moves
${REGISTRY}/${ORG}/${APP}:worker
${REGISTRY}/${ORG}/${APP}:v1.4.2-worker
```

Second, the deploy script accepts `--rollback v1.4.2`. It writes a compose override pinning the immutable tags, pulls, recreates, and runs the same health gate, without migrations.

```bash
cd /srv/app && ./deploy.sh --rollback v1.4.2
```

The stack stays pinned until the next normal deploy, which removes the override file. Verify a rollback once per quarter, not during an incident.

## Failure modes

- **Site stuck in maintenance mode.** Cause: a step between `artisan down` and `artisan up` failed without a trap. Fix: use the trap in this page, and keep `php artisan up` in the runbook.
- **Deploy reports success but nothing changed.** Cause: compose filename mismatch (`compose.yml` shipped where the stack uses `compose.yaml`), or CI never moved the tag. Fix: digest check plus the no-op warning, and verify the tag exists in the registry.
- **`/up` returns 200 but users see 500s.** Cause: the health route does not touch the database, cache, or storage. Fix: make `/up` check dependencies, and add a post-deploy exception check for critical routes.
- **Storage permission failure on first boot.** Cause: the deploy user cannot `chown` to uid 82, so framework directories stay unwritable. Fix: the throwaway root container block above, verified by a second container running as 82.
- **Migration fails and blocks every future deploy.** Cause: a non-idempotent migration partially applied DDL and was never recorded. Fix: guard DDL with `hasColumn`, backfill with chunks, never edit the database by hand.
- **Rollback missing.** Cause: only a mutable tag was published. Fix: publish immutable tags from day one, before you need them.

## Checklist

- [ ] `compose.yaml` and `deploy.sh` are versioned in the repo and shipped by CI
- [ ] `.env` is operator-managed, mode 600, and validated before maintenance mode
- [ ] Registry credentials arrive as `.registry.env` mode 600
- [ ] Deploy key is forced-command restricted
- [ ] Mutable and immutable image tags exist for both roles
- [ ] Storage permissions are prepared from a root container and verified
- [ ] Maintenance mode has a trap that restores the app
- [ ] `/up` is dependency-aware and gated with retries
- [ ] Digest change is asserted, no-op deploys are loud
- [ ] `--rollback TAG` works and is tested once per quarter
- [ ] Deploy notifications go somewhere a human reads
- [ ] The runbook documents deploy, rollback, and restore
