By the end of this page you have one deploy script that works for every app, fails loudly, restores the site when a step fails, refuses to deploy a half-configured stack, and gives you a one-line rollback.
Everything before this page produced an image. This chapter is about the twenty seconds where that image becomes the thing users hit, which is the twenty seconds most incidents live in.
The server layout
One directory per app, owned by the deploy user. Nothing else on the box knows about it.
/srv/app/
├── compose.yaml # shipped by CI on every deploy
├── deploy.sh # shipped by CI on every deploy
├── .env # operator-managed, mode 600, never in git
├── .registry.env # registry credentials, mode 600, written by CI
├── .storage/ # bind-mounted app storage (logs, app, framework)
│ ├── logs/
│ ├── app/
│ └── framework/
├── .postgres-db/ # bind-mounted database data
├── backups/ # database dumps (or a volume)
└── compose.rollback.yaml # generated only while a rollback is pinned
The app's git repository is not needed on the host at all. CI ships the two files it owns. The host owns config and data.
Why the deploy is pull-only
The deploy script never runs docker build, composer install, npm ci, or git pull. It pulls a prebuilt image, replaces containers, migrates, and verifies.
This is not a style preference. It means:
- the bytes that ran in tests are the bytes that run in production;
- the production host needs no compilers, no Node, and no Composer;
- a release takes seconds, not minutes, so rollback is cheap;
- a failed build cannot take the site down, because it happens somewhere else.
If a deploy needs a rebuild, that is a CI trigger, not a deploy step.
The sequence
Every script in this guide follows the same contract. Read the order carefully, because the order is the safety.
- Validate config before touching anything. Missing
APP_KEYor a database password should fail before maintenance mode, not halfway through. - Capture the running image digest. This makes a no-op deploy visible.
- Log in to the registry and pull. Pull before maintenance mode so a registry outage does not take the site down for nothing.
- Prepare storage bind mounts. The container runs as a non-root user; the host deploy user cannot
chownto it. Do it from a throwaway root container. - Enable maintenance mode. Short, honest downtime beats a half-broken rollout on a single box. Skip it only for stateless apps or multi-replica rolling updates.
- Recreate the app containers.
--force-recreateon the long-running processes, so Octane or the queue worker actually boots the new code. - Run migrations.
--force, forward-only, never on rollback. - Run the app's post-deploy hook. Cache warm, search index, backfill. A file in the repo (
scripts/post-deploy.sh) keeps this generic. - Leave maintenance mode with retries. A failed
artisan upmust never be permanent. - Probe
/upfrom inside the container, with retries. Health is a dependency-aware route, not a TCP check. - Warn if the image digest did not change. A deploy that changed nothing should say so.
- Notify and exit non-zero on failure. A silent failure is how backups stay broken for a year.
The failure invariant is the important part: if anything between step 5 and step 9 fails, a trap takes the site out of maintenance mode and the script exits non-zero, so CI shows red.
The script
This is the canonical version. Copy it into scripts/deploy.sh in every repo, adjust REQUIRED_VARS and the health URL, and never fork it per app without a reason.
#!/usr/bin/env bash
# deploy.sh - pull prebuilt images and deploy this app on its host.
# Runs ON the host, invoked by CI (or by a forced-command wrapper).
set -euo pipefail
APP_DIR="${APP_DIR:-/srv/app}"
COMPOSE_FILE="${COMPOSE_FILE:-$APP_DIR/compose.yaml}"
REGISTRY="${REGISTRY:-${REGISTRY_HOST}}"
IMAGE_REPO="${IMAGE_REPO:-${REGISTRY}/${ORG}/${APP}}"
HEALTH_PATH="${HEALTH_PATH:-/up}"
SERVICES=(web worker scheduler)
NTFY_URL="${NTFY_URL:-}"
cd "$APP_DIR"
# ---------------------------------------------------------------- helpers
red() { printf '\033[0;31m%s\033[0m\n' "$*" >&2; }
green() { printf '\033[0;32m%s\033[0m\n' "$*"; }
warn() { printf '\033[1;33mWARNING: %s\033[0m\n' "$*" >&2; }
notify() {
local title="$1" message="$2" priority="${3:-low}"
[ -n "$NTFY_URL" ] || return 0
curl -fsS --max-time 10 -H "Title: $title" -H "Priority: $priority" \
-d "$message" "$NTFY_URL" >/dev/null 2>&1 || true
}
# ---------------------------------------------------------------- preflight
REQUIRED_VARS=(APP_KEY APP_ENV APP_URL APP_DOMAIN DB_CONNECTION DB_HOST DB_PORT
DB_DATABASE DB_USERNAME DB_PASSWORD REDIS_HOST REDIS_PORT
QUEUE_CONNECTION CACHE_STORE)
[ -f "$COMPOSE_FILE" ] || { red "missing $COMPOSE_FILE"; exit 1; }
[ -f .env ] || { red "missing $APP_DIR/.env"; exit 1; }
set -a
# shellcheck disable=SC1091
. ./.env
set +a
missing=()
for var in "${REQUIRED_VARS[@]}"; do
[ -n "${!var:-}" ] || missing+=("$var")
done
if [ "${#missing[@]}" -gt 0 ]; then
red "missing or empty required vars in .env:"
printf ' - %s\n' "${missing[@]}" >&2
exit 1
fi
if [ "$APP_ENV" != "production" ]; then
red "APP_ENV must be production, found '$APP_ENV'"
exit 1
fi
if grep -qE '^APP_KEY=base64:dummy' .env; then
red "APP_KEY is the build-time dummy. Refusing to deploy."
exit 1
fi
MODE="deploy"
ROLLBACK_TAG=""
case "${1:-}" in
--check) MODE="check" ;;
--rollback) MODE="rollback"; ROLLBACK_TAG="${2:-}"
[ -n "$ROLLBACK_TAG" ] || { red "--rollback needs a tag"; exit 2; } ;;
"") ;;
*) red "unknown argument: $1"; exit 2 ;;
esac
compose() { docker compose -f "$COMPOSE_FILE" "$@"; }
# Digest of the currently running web container, to detect no-op deploys.
current_image() {
docker inspect --format '{{.Image}}' "$(compose ps -q web 2>/dev/null)" 2>/dev/null || true
}
# ---------------------------------------------------------------- check mode
if [ "$MODE" = "check" ]; then
green "env OK (${#REQUIRED_VARS[@]} required vars present)"
compose config -q && green "compose config OK"
compose config --images | sed 's/^/ /'
exit 0
fi
# ---------------------------------------------------------------- rollback
if [ "$MODE" = "rollback" ]; then
green "rolling back to $ROLLBACK_TAG (migrations are NOT run)"
cat > compose.rollback.yaml <<EOF
services:
web:
image: ${IMAGE_REPO}:${ROLLBACK_TAG}-web
worker:
image: ${IMAGE_REPO}:${ROLLBACK_TAG}-worker
scheduler:
image: ${IMAGE_REPO}:${ROLLBACK_TAG}-worker
EOF
COMPOSE_FILE="$APP_DIR/compose.yaml"
docker compose -f "$COMPOSE_FILE" -f compose.rollback.yaml pull
docker compose -f "$COMPOSE_FILE" -f compose.rollback.yaml up -d --remove-orphans
for attempt in 1 2 3 4 5 6 7 8 9 10; do
if compose exec -T web sh -c "curl -fsS -o /dev/null http://127.0.0.1:80${HEALTH_PATH}"; then
green "/up OK after rollback"
notify "Rollback complete" "app rolled back to ${ROLLBACK_TAG}" default
exit 0
fi
sleep 3
done
red "rollback started but /up never recovered"
compose logs --tail=60 web >&2 || true
notify "Rollback failed" "app rollback to ${ROLLBACK_TAG} did not pass /up" high
exit 1
fi
# ---------------------------------------------------------------- deploy
DEPLOY_START=$(date +%s)
notify "Deploy started" "APP deploy started" low
cleanup() {
local rc=$?
if [ "$rc" -ne 0 ]; then
red "deploy failed (rc=$rc), restoring maintenance mode"
compose exec -T web php artisan up >/dev/null 2>&1 || true
compose logs --tail=80 web >&2 || true
notify "Deploy failed" "deploy failed with rc=$rc; app restored" high
fi
exit "$rc"
}
trap cleanup EXIT
# 1. registry login (credentials shipped by CI, mode 600)
if [ -f .registry.env ]; then
set -a
# shellcheck disable=SC1091
. ./.registry.env
set +a
fi
if [ -n "${FORGEJO_REGISTRY_TOKEN:-}" ]; then
printf '%s' "$FORGEJO_REGISTRY_TOKEN" |
docker login "$REGISTRY" -u "${FORGEJO_REGISTRY_USER:?}" --password-stdin
fi
green "[1/9] pulling images"
before="$(current_image)"
compose pull
green "[2/9] preparing storage bind mounts"
for image in "${IMAGE_REPO}:web" "${IMAGE_REPO}:worker"; do
docker run --rm --user 0 -v "$APP_DIR/.storage:/app/storage" \
--entrypoint sh "$image" -c \
'mkdir -p /app/storage/logs /app/storage/app/public /app/storage/framework && chown -R 82:82 /app/storage'
done
green "[3/9] enabling maintenance mode"
compose exec -T web php artisan down --retry=60 2>/dev/null ||
warn "app not running yet, skipping maintenance mode"
green "[4/9] recreating containers"
compose up -d --remove-orphans --force-recreate "${SERVICES[@]}"
green "[5/9] running migrations"
compose exec -T web php artisan migrate --force
green "[6/9] running post-deploy hook"
if [ -x scripts/post-deploy.sh ]; then
./scripts/post-deploy.sh
else
warn "no scripts/post-deploy.sh, skipping cache warm/backfill"
fi
green "[7/9] leaving maintenance mode"
up_attempt=1
while [ "$up_attempt" -le 6 ]; do
if compose exec -T web php artisan up; then
break
fi
warn "artisan up failed (attempt ${up_attempt}/6), retrying"
up_attempt=$((up_attempt + 1))
sleep 5
done
[ "$up_attempt" -le 6 ] || { red "app may still be in maintenance mode"; exit 1; }
green "[8/9] probing ${HEALTH_PATH}"
status="000"
for attempt in 1 2 3 4 5 6 7 8 9 10; do
status="$(compose exec -T web sh -c \
"curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:80${HEALTH_PATH}")" || status="000"
case "$status" in 2*|3*) break ;; esac
sleep 3
done
case "$status" in
2*|3*) green "health gate passed ($status)" ;;
*) red "health gate failed (last HTTP $status)"; exit 1 ;;
esac
green "[9/9] reporting"
after="$(current_image)"
if [ -n "$before" ] && [ "$before" = "$after" ]; then
warn "image digest unchanged ($after), this deploy changed nothing"
fi
trap - EXIT
duration=$(( $(date +%s) - DEPLOY_START ))
compose ps
green "deployed in ${duration}s"
notify "Deploy complete" "APP deployed in ${duration}s" default
A few details that are easy to miss:
- The digest check matters. If CI shipped the wrong compose filename, or the tag did not move, the digest is identical and the script says so instead of reporting success.
artisan upruns both in the trap and in step 7 on purpose. The trap covers failures between them; the explicit retry covers a slow database. Both are cheap.- The rollback branch deliberately does not run migrations. Migrations are forward-only in this design. If a release needs a destructive migration, the rollback plan is a database restore, and that is a separate documented operation.
- The storage block runs before maintenance mode and before recreate, so the app never starts against an unwritable framework directory.
Handling CI invocation
CI ships exactly two files and then calls the script.
- name: deploy
when:
- event: tag
ref: 'refs/tags/v*'
commands:
- apk add --no-cache openssh-client
- mkdir -p ~/.ssh && chmod 700 ~/.ssh
- printf '%s\n' "$DEPLOY_SSH_KEY" > ~/.ssh/id_ed25519
- chmod 600 ~/.ssh/id_ed25519
- scp -O -q -i ~/.ssh/id_ed25519 compose.yaml "deploy@${DEPLOY_HOST}:${APP_DIR}/compose.yaml"
- scp -O -q -i ~/.ssh/id_ed25519 scripts/deploy.sh "deploy@${DEPLOY_HOST}:${APP_DIR}/deploy.sh"
- scp -O -q -i ~/.ssh/id_ed25519 .registry.env "deploy@${DEPLOY_HOST}:${APP_DIR}/.registry.env"
- ssh -i ~/.ssh/id_ed25519 "deploy@${DEPLOY_HOST}" "cd ${APP_DIR} && chmod +x deploy.sh && ./deploy.sh"
Better: lock the deploy key down with a forced command on the host, so a leaked CI secret can only run the deploy, not open a shell.
# /home/deploy/.ssh/authorized_keys
command="cd /srv/app && ./deploy.sh",no-agent-forwarding,no-port-forwarding,no-pty ssh-ed25519 AAAA... ci-deploy
And on the host side, .registry.env must be mode 600. CI writes it, scp preserves 600 only if it was created that way, so do install -m 600 or chmod in the script.
Zero downtime, honestly
On a single cheap VPS with one database and one web container, true zero downtime is mostly a story people tell. The honest options, in order of cost:
- Maintenance mode (this guide). 10 to 40 seconds of 503 with
Retry-After. Simple, safe, and fine for a blog, a directory, or an internal tool. Cloudflare can serve a cached version of the page during the window if you enable "cache everything" with a bypass for cookies. - Two replicas behind Traefik and a rolling recreate.
docker compose up -d --scale web=2with Traefik health checks, then replace one at a time. Works only if the app is truly stateless and any one replica can serve while the other boots, and it doubles memory on a small box. - Separate release directories and an atomic Traefik switch. More moving parts, one more thing to get wrong. Worth it only for a revenue-critical service.
Do not claim zero downtime because the container starts fast. The database migration and the maintenance window are where the outage lives.
Rollback
Two rules make rollback possible.
First, CI pushes two tags per build: a mutable pointer used by normal deploys, and an immutable tag used by rollbacks.
${REGISTRY}/${ORG}/${APP}:web # moves on every release
${REGISTRY}/${ORG}/${APP}:v1.4.2-web # never moves
${REGISTRY}/${ORG}/${APP}:worker
${REGISTRY}/${ORG}/${APP}:v1.4.2-worker
Second, the deploy script accepts --rollback v1.4.2. It writes a compose override pinning the immutable tags, pulls, recreates, and runs the same health gate, without migrations.
cd /srv/app && ./deploy.sh --rollback v1.4.2
The stack stays pinned until the next normal deploy, which removes the override file. Verify a rollback once per quarter, not during an incident.
Failure modes
- Site stuck in maintenance mode. Cause: a step between
artisan downandartisan upfailed without a trap. Fix: use the trap in this page, and keepphp artisan upin the runbook. - Deploy reports success but nothing changed. Cause: compose filename mismatch (
compose.ymlshipped where the stack usescompose.yaml), or CI never moved the tag. Fix: digest check plus the no-op warning, and verify the tag exists in the registry. /upreturns 200 but users see 500s. Cause: the health route does not touch the database, cache, or storage. Fix: make/upcheck dependencies, and add a post-deploy exception check for critical routes.- Storage permission failure on first boot. Cause: the deploy user cannot
chownto uid 82, so framework directories stay unwritable. Fix: the throwaway root container block above, verified by a second container running as 82. - Migration fails and blocks every future deploy. Cause: a non-idempotent migration partially applied DDL and was never recorded. Fix: guard DDL with
hasColumn, backfill with chunks, never edit the database by hand. - Rollback missing. Cause: only a mutable tag was published. Fix: publish immutable tags from day one, before you need them.
Checklist
-
compose.yamlanddeploy.share versioned in the repo and shipped by CI -
.envis operator-managed, mode 600, and validated before maintenance mode - Registry credentials arrive as
.registry.envmode 600 - Deploy key is forced-command restricted
- Mutable and immutable image tags exist for both roles
- Storage permissions are prepared from a root container and verified
- Maintenance mode has a trap that restores the app
-
/upis dependency-aware and gated with retries - Digest change is asserted, no-op deploys are loud
-
--rollback TAGworks and is tested once per quarter - Deploy notifications go somewhere a human reads
- The runbook documents deploy, rollback, and restore
Proof you are done
- - ./deploy.sh --check
- - ./deploy.sh
- - curl -fsS -o /dev/null -w '%{http_code}\n' "https://app.example.com/up"
- - curl -s -o /dev/null --max-time 5 "https://${VPS_IP}/" -H "Host: app.example.com" || echo "origin blocked, good"
- - ./deploy.sh
- - docker image ls "${REGISTRY}/${ORG}/${APP}" | head