Monitoring
Health checks
| Check | Command or address | Expected |
|---|---|---|
| API | https://bemba.ucp-pch.org/api/health | 200 |
| Application | https://bemba.ucp-pch.org/ | 200 |
| Documentation | https://docubemba.ucp-pch.org/ | 200 |
| Containers | docker ps --format '{{.Names}}\t{{.Status}}' | all Up, healthy when a check exists |
| E-mail queue | docker exec pch_backend php bin/console messenger:failed:show | empty |
| Disk space | df -h / | under 80 % |
| Memory | free -m | comfortable "available" |
From inside the container (bypassing Traefik):
docker exec pch_backend curl -s http://127.0.0.1:8000/api/health
Metrics
Production ships Prometheus exporters, bound to 127.0.0.1 (not exposed to the Internet):
| Exporter | Port | Measures |
|---|---|---|
| cAdvisor | 8080 | CPU, memory, network per container |
| node-exporter | 9100 | system (disk, load, memory) |
| postgres-exporter | 9187 | pch_sig database (connections, size, locks) |
| redis-exporter | 9121 | cache |
Prometheus and Grafana do not run on the servers: they run on the admin workstation (monitoring/ folder of the repository, Grafana on port 3001). To reach the exporters, open SSH tunnels:
ssh -N -L 8080:127.0.0.1:8080 -L 9100:127.0.0.1:9100 -L 9187:127.0.0.1:9187 \
-L 9121:127.0.0.1:9121 -L 3100:127.0.0.1:3100 pch-prod
then point the targets in monitoring/prometheus/prometheus.yml to localhost and Grafana's Loki data source to http://localhost:3100.
note
The Prometheus configuration in the repository still targets the servers' public addresses, which no longer answer since the ports were bound to 127.0.0.1. Use the tunnels above.
Application errors
Sentry receives frontend and backend errors, with context. It is the first place to look after a user report.
Warning signs
| Signal | Common cause | Action |
|---|---|---|
Every request returns 500 right after a cache:clear | cache rebuilt by root | recreate the backend container (see Troubleshooting) |
https://… shows a "TRAEFIK DEFAULT CERT" certificate | DNS missing when the certificate was requested | check DNS, then docker restart pch_traefik |
Growing failed queue | OVH SMTP down or password changed | see Configuration, E-mails section |
| Disk full | logs, Docker images | docker image prune -f, docker builder prune -af |
| KoboToolbox returns 502 (pre-production) | still starting (several minutes) | wait, then docker ps --filter name=kobofe |
| Survey Solutions stopped | restarted automatically by check-suso.sh every 5 minutes | read ~/suso-healthcheck.log |