Scenario
An e-commerce platform running MySQL 5.7 saw daily full backups taking over 8 hours, with weekly restore drills exceeding 12 hours, severely impacting backup windows and RTO. Similarly, a PostgreSQL 12 database using streaming replication experienced high CPU spikes during pg_dump logical backups, causing primary query latency.
Common Symptoms
- Slow Backup: mysqldump or pg_dump runs far longer than expected, disk I/O near 100%, high CPU load.
- Slow Restore: Import from backup is much slower than anticipated, especially with large tables or many indexes.
- Log Explosion: Binary logs (MySQL) or WAL logs (PostgreSQL) grow abnormally, leading to disk space issues.
Diagnosis
MySQL
- Check backup command options:
mysqldump --single-transaction --quick --compress --skip-lock-tablescan speed up logical backup. - Use
SHOW ENGINE INNODB STATUSto look for long transactions blocking backup. - Monitor disk performance:
iostat -x 1for %util and await values to identify bottlenecks. - Analyze backup queries using general log or
pt-query-digest.
PostgreSQL
- Verify pg_dump parallelism:
pg_dump -j 4for parallel dump (PG 9.4+). - Check WAL archiving: compare
pg_current_wal_lsn()with archive location. - Use
pg_stat_activityto see if backup process is blocked by other queries. - Tune parameters:
wal_buffers,max_wal_size,checkpoint_completion_targetaffect write performance.
Example Commands
MySQL Optimized Backup
# Physical backup using innobackupex (faster)
innobackupex --user=backup --password --parallel=4 --no-timestamp /backup/
# Throttle backup rate to avoid impact
pvb -i 300m -r 100m | mysqldump ... > dump.sql
PostgreSQL Optimized Backup
# Parallel pg_dump (only PG 9.4+)
pg_dump -j 4 -Fd -f /backup/dump_dir dbname
# Physical backup with pg_basebackup
pg_basebackup -D /backup -X stream -P -v -z -Z 6
Risk Controls
- Avoid DDL during backup (MySQL:
lock_wait_timeout, PG:lock_timeout). - Limit IOPS:
ionice -c2 -n7to lower backup priority. - Monitor progress: MySQL
PROCESSLIST’sTimecolumn; PGpg_stat_progress_*views. - Use transaction snapshot isolation: MySQL
--single-transaction, PG default repeatable read.
Rollback Plan
Always retain the last full backup and incremental backups. If a restore fails:
1. Stop the restore process to avoid overwriting data.
2. Reload from cold backup or replica.
3. Verify backup file integrity: mysqlcheck --all-databases or pg_checksums.
Verification
- After restore, run
SELECT COUNT(*)to compare row counts with original table. - Use
pt-table-checksum(MySQL) orpg_verifybackup(PG) to check consistency. - Check recent transaction timestamps for sanity.
When to Submit an OpsGlobal Ticket
- Backup/restore performance issues persist beyond 2 hours.
- Data corruption or loss is suspected.
- Expert tuning of database parameters or backup strategy is needed.
- Backup tool version incompatibility causes errors.
Use cases
Useful for teams handling Database issues and needing a clear troubleshooting and delivery workflow.
Problem background
This article dives into real-world performance issues during backup and recovery of MySQL and PostgreSQL databases, offering diagnostic steps, optimized commands, and safety controls for SRE teams.
Troubleshooting steps
Confirm impact and recent changes, collect logs, configuration and metrics, then apply fixes from low to high risk.
Command examples
Replace sample resource names with real values and store passwords, tokens and keys in environment variables.
Risks
Before production changes, confirm backups, access boundaries, change windows and rollback paths.
Rollback plan
Keep original configuration and release versions; roll back config, images or database changes if metrics degrade.
Deliverables
Root-cause notes, key commands, remediation steps, verification results and follow-up recommendations.
Need help with a similar technical issue?
If your servers, Kubernetes, Docker, CI/CD, databases or monitoring systems have similar issues, submit logs and config files for remote diagnosis.