Most Postgres backup advice is a checklist: run pg_dump nightly, ship it to S3, call it done. GitLab followed a version of that checklist in January 2017, and it still lost roughly six hours of production data because every actual backup mechanism they had either wasn't working or covered the wrong thing. The company's own postmortem is one of the most detailed public accounts of a database incident ever published, and it's a better guide to a real Postgres backup strategy than any best-practices list, because it shows exactly which parts of a "complete" setup quietly stop working.

What actually happened

On January 31, 2017, an engineer took a routine LVM snapshot of GitLab.com's production database at 17:20 UTC and loaded it into staging, for a completely unrelated project testing connection pooling. That snapshot, taken for a reason that had nothing to do with disaster recovery, turned out to be the only thing standing between GitLab and total data loss a few hours later.

Increased load caused the database's secondary replica to fall behind on replication, and the replication broke outright: GitLab wasn't using WAL archiving, so the write-ahead log segments the secondary needed had already been deleted from the primary. Fixing this meant wiping the secondary's data directory and re-running pg_basebackup to resync it from scratch, a normal, if tedious, recovery step.

It didn't go smoothly. pg_basebackup hung with no useful output for hours, through a max_wal_senders limit that blocked replication connections, a max_connections setting of 8,000 that made Postgres refuse to restart once they tried raising it, and finally a set of stray files left behind by earlier failed attempts, a gotcha documented in neither GitLab's own runbooks nor Postgres's official docs.

Trying to clear those stray files, an engineer wiped the database data directory again, certain he was on the secondary. He was on the primary. He killed the process within a second or two of realizing the mistake. By then, around 300GB of data was already gone.

Why the backup system didn't save them

At this point, replication was broken and both database hosts had had their data wiped. There was no live host left to fail over to. That's exactly the moment backups exist for, and GitLab had three separate mechanisms in place. None of them worked.

pg_dump to S3, run every 24 hours. When engineers went looking for a recent backup, the S3 bucket was empty. The backup script had been running pg_dump version 9.2 against a database running Postgres 9.6, a major-version mismatch that makes pg_dump fail outright. Nobody had checked whether the backups were actually landing, so the failure had been silent for an unknown stretch of time.

Azure disk snapshots, also every 24 hours. These existed and ran fine, but they backed up the NFS servers storing Git repository data, a different dataset entirely from the Postgres database that had just been wiped.

Streaming replication between the primary and secondary. This existed purely for failover between two live hosts, and it had just gone down as part of the same incident that caused the outage, so it had nothing left to fail over to.

Three mechanisms, three different reasons none of them applied when the moment actually arrived: silently broken, aimed at the wrong data, and structurally unable to help with this specific failure. GitLab is a company with a dedicated infrastructure team and a published disaster recovery process, and it still took an unrelated coincidence to avoid losing everything.

The snapshot that actually saved them, and its own hidden cost

What recovered GitLab's data was the LVM snapshot from 17:20 UTC, the one taken hours earlier for a project that had nothing to do with backups. It wasn't part of the automated daily backup rotation. It existed because an engineer happened to need a copy of production for an unrelated test.

Restoring from it still took 18 hours. The snapshot lived in a staging environment running on Azure classic storage without the premium tier, a deliberate cost-saving choice, and those disks were throttled to roughly 60 Mbps. With the network and CPUs both sitting idle, the entire delay came down to raw disk speed. Nobody had ever needed to move that much data off staging quickly before, so nobody had noticed the throttling would matter.

The total damage: roughly six hours of data lost, between 17:20 and 00:00 UTC, at least 5,000 projects, 5,000 comments, and around 700 user accounts gone for good. Self-managed GitLab installations weren't affected, only the hosted GitLab.com service was.

What this means for your own Postgres backup strategy

GitLab's own stated conclusion wasn't to add more approval steps or restrict which engineers could run destructive commands. They explicitly rejected that as the fix, arguing that an environment where mistakes are survivable and recovery is fast beats one that just tries to prevent every possible mistake. For a solo developer or a small team running Postgres without a dedicated ops function, that framing matters more than any specific tool choice.

A few things fall directly out of this incident that a generic "how to back up Postgres" guide won't tell you. Pin and actually test your backup tooling's version against your real database version, a mismatch can fail silently for months. Run an actual restore on a real schedule as the check that a backup exists, a cron job that appears to run is exactly how GitLab's own S3 bucket sat empty without anyone noticing.

Replication protects against a healthy host failing over to another one; it does nothing once the data itself is destroyed, so it belongs in a different column of your plan than backups do. And know how long a real restore actually takes before you need it, an 18-hour recovery window is a very different business decision than a two-hour one.

If you're running Postgres on a single box, you don't have GitLab's infrastructure team, and you also don't have their scale of complexity. A pg_dump cron job and an occasional manual restore test will catch most of what broke here, if you actually run the restore test. A company with three separate backup systems still nearly lost everything; the real lesson is that a backup only counts once you've proven you can restore from it under pressure.

Read GitLab's full postmortem if you're setting up backups for anything you'd actually miss. It's more useful than most vendor documentation on the same topic, because it shows what a real failure looks like from the inside, the part vendor documentation almost never covers. The same discipline applies once your setup is stable enough to need zero-downtime schema changes: test the risky operation before you're forced into it under pressure.