Advanced
The logs that filled the disk
Twice this month the shop stopped taking orders in the middle of the night: PostgreSQL could not write any more because the disk was full. docker system df showed nothing unusual. The culprit is the payment webhooks service, which logs at debug level for an ongoing investigation. Keep those logs, but make sure no container can ever fill the disk with its logs again.
The server has a 40 GB disk, and the database already takes 18 GB. The payments team needs the debug logs for two more weeks: lowering LOG_LEVEL is not an option. The test runs docker compose up -d --build and places an order every minute for 24 hours.
What your server must hold
- Availability ≥ 99.5 %
- Disk usage at its peak < 70 %
- disk.unboundedLogs = 0
- app.logsDisabled = 0