Skip to main content

Troubleshooting

Every case below has already happened in production or pre-production.

Quick diagnosis​

cd ~/pch-sig
F=docker-compose.prod.yml          # or docker-compose.preprod.yml
docker compose -f $F ps
docker exec pch_backend curl -s http://127.0.0.1:8000/api/health
docker logs pch_backend --since 15m 2>&1 | grep -iE "error|exception" | tail -20
df -h / && free -m

Every request returns 500​

After a cache:clear run with docker exec (root). The cache is rebuilt owned by root and php-fpm (www-data) can no longer write the Doctrine proxies:

Your proxy directory "/app/var/cache/prod/doctrine/orm/Proxies" must be writable

The command had printed [OK], and any check run as root works: it is misleading.

docker compose -f $F up -d --no-deps backend        # the entrypoint clears the cache and resets permissions
# or
docker exec pch_backend sh -c 'chown -R www-data:www-data /app/var'

Rule: every console command runs with docker exec -u www-data, and every check runs as the real user.

500 at sign-in ("Unable to create a signed JWT"): the private JWT key is not readable by www-data, or the .env.prod passphrase does not open it. The keys are mounted read-only from ~/pch-sig/backend/config/jwt: fix permissions on the host (owner 82:82, which is the image's www-data, and mode 600 for the private key), not inside the container. See JWT authentication.

The deployed fix is not active​

A failed Docker build leaves the old container running, healthy. Look inside the container for a string specific to the new code (docker exec pch_backend grep -c "…" /app/src/…) and read ~/build-backend.log. The build runs detached (nohup) so that an SSH drop does not stop it.

Unexpected containers (pch-sig-database-1, pch-sig-mailer-1)​

docker compose was run without -f: it started the Symfony skeleton (compose.yaml). Stop them with docker compose -f compose.yaml down, then restart the application with the right -f.

Backend keeps restarting​

docker logs pch_backend --tail 50
  • exec /usr/local/bin/docker-entrypoint.sh: no such file or directory: Windows line endings in the script. sed -i 's/\r$//' backend/docker/*.sh, then rebuild.
  • Unable to read the "/app/.env" environment file: backend/.env was missing at build time.
  • Failed migration: the SQL message is in the log; fix the migration and redeploy.

KeyError: 'ContainerConfig'​

Compose v1 (docker-compose) with Docker Engine 28. Install the v2 plugin (docker-compose-plugin) and use docker compose.

Invalid certificate ("TRAEFIK DEFAULT CERT")​

The name's DNS did not exist when Traefik requested the certificate. Check nslookup <name>, then docker restart pch_traefik to retry. Read docker logs pch_traefik | grep -i acme.

E-mails not received​

  1. docker exec pch_backend php bin/console messenger:failed:show: failed messages and reason.
  2. In pre-production this is expected: SMTP is unreachable on purpose.
  3. 535 Authentication failed: the password of noreply@ucp-pch.org was changed at OVH without updating .env.prod.

KoboToolbox (pre-production): its e-mails go through the production relay. docker logs smtp_relay on production shows status=sent or the reason. If the relay once started with a wrong password, it keeps it as a duplicate: docker compose -f docker-compose.yml down -v && … up -d in ~/smtp-relay.

Blank screen or old version after a deployment​

  • Force a reload (Ctrl+Shift+R); the service worker updates on the next reload.
  • Check that the served files contain the change (search every chunk of frontend/build/static/js).
  • Browser console: a resource blocked by the CSP shows as "Refused to connect"; add the domain to nginx/frontend.conf if the call is legitimate.

A user sees no data​

It is almost never a bug: the account is attached to no institution or no project (partitioning). Check under Settings → Users. An account can be authorised (permissions) without being attached.

The Bemba assistant does not cite the documentation​

docker exec -u www-data pch_backend php bin/console app:assistant:indexer --etat

Empty index: the documentation build is not mounted (/app/documentation-build) or indexing has not run. Run app:assistant:indexer again.

Survey Solutions no longer syncs​

The connection is read from the survey_solutions_config table (not from environment variables). Check the stored password, then Test connection from the sync screen.

Disk full​

docker system df
docker image prune -f && docker builder prune -af
du -sh ~/backups

What to include in a report​

Exact time, user concerned, screen, message shown, and the matching excerpt of docker logs pch_backend (without passwords or tokens).