Backup

Overview

Data	Mechanism	Frequency	Retention	Location
Production database (pch_sig)	continuous PostgreSQL replication to the recovery host	continuous	—	axone-dr
Production database	encrypted restic snapshots on the recovery host	daily	per restic policy	axone-dr
Production logs (loki_data volume)	incremental rsync + restic	daily	Loki keeps one year	axone-dr
Pre-production database	compressed pg_dump (~/scripts/backup-database.sh)	02:00	30 days	~/backups on the 220 server
Pre-production GitLab	gitlab-backup create + archive of /etc/gitlab (~/scripts/backup-gitlab.sh)	02:30	14 days	~/backups/gitlab on the 220 server
Source code	Git repositories	on every push	—	gitlab.com and gitlab.ucp-pch.org

The recovery host (axone-dr) is run by Axone; it connects to production through the pch_pg_dr_proxy container.

Open points

• Uploaded files (backend_uploads volume: photos, attachments, documents): their off-site copy still has to be confirmed with the team that runs the recovery host.
• Backups of the 220 server (pre-production, GitLab) stay on that server: they are not yet copied off-site.

Manual database backup

Before a risky operation (heavy migration, bulk import):

# Production
docker exec pch_postgres sh -c 'pg_dump -U "$POSTGRES_USER" -d "$POSTGRES_DB"' | gzip > ~/backups/pch_sig_$(date +%Y%m%d_%H%M%S).sql.gz

# Pre-production (prefixed container name)
PG=$(docker ps --filter name=pch_postgres --format '{{.Names}}' | grep -v exporter | head -1)
docker exec "$PG" sh -c 'pg_dump -U "$POSTGRES_USER" -d "$POSTGRES_DB"' | gzip > ~/backups/pch_sig_$(date +%Y%m%d_%H%M%S).sql.gz

Check that the file is not empty (ls -lh ~/backups | tail -3).

Uploaded files backup

docker run --rm -v pch-sig_backend_uploads:/data:ro -v ~/backups:/out alpine \
  tar -czf /out/uploads_$(date +%Y%m%d).tar.gz -C /data .

(The exact volume name is given by docker volume ls | grep uploads.)

Pre-production backups

0 2 * * *  /home/ubuntu/scripts/backup-database.sh >> /home/ubuntu/backups/cron.log 2>&1
30 2 * * * /home/ubuntu/scripts/backup-gitlab.sh   >> /home/ubuntu/backups/cron.log 2>&1

The GitLab backup produces two files a day, both required for a restore: the application archive (*_gitlab_backup.tar) and the configuration (gitlab-config_*.tar.gz, which contains gitlab-secrets.json). Details in docs/GITLAB_SERVEUR.md.

Check

ls -lh ~/backups | tail -5                       # recent, non-empty dumps
tail -20 ~/backups/backup.log                     # pre-production script log
tail -20 ~/backups/gitlab-backup.log

A backup is only reliable if its restore has been tested: see Restore.

What must never be lost

• ~/pch-sig/.env.prod and ~/pch-sig/backend/.env (secrets), backend/config/jwt/ (JWT keys): their content is in the secrets manager.
• ~/smtp-relay/.env (SMTP relay), ~/kobo-install/ (Kobo configuration), ~/gitlab (GitLab configuration).