pgbackrest/pgbackrest: Reliable PostgreSQL Backup & Restore · GitHub

TL;DR: pgBackRest is no longer maintained. If you fork pgBackRest, please choose a new name for your project.

After much thought, I have decided to stop working on pgBackRest. I did not come to this decision lightly. pgBackRest has been my passion project for the past thirteen years, and I was lucky to have corporate sponsorship for much of this time, but there were also many late nights and weekends as I worked to make pgBackRest the project it is today, with the help of many contributors. Every open-source developer knows what I mean and how much of your life is dedicated to a particular project.

Since Crunchy Data was sold, I have been maintaining pgBackRest and looking for a position that would allow me to continue the work, but so far I have not succeeded. Likewise, my efforts to secure sponsorship also fall far short of the efforts required to make the project viable.

Like everyone else, I need to earn a living, and the range of pgBackRest-related roles is very limited. I can consider a variety of opportunities now, but they won’t give me time to work on pgBackRest, which requires a fair amount of time for maintenance, bug fixes, PR reviews, responding to issues, etc. This doesn’t even include time to write new features, which I really love doing. Rather than doing the job poorly and/or sporadically, I think it is more reasonable to stop it strictly.

I think pgBackRest will be abandoned at some point, but it will be a new project with new maintainers, and they will need to build trust just like we did.

Again, many thanks to all the pgBackRest contributors over the years. it was a pleasure working with you!

pgBackRest is a reliable backup and restore solution for PostgreSQL that scales seamlessly to the largest databases and workloads.

pgBackRest v2.58.0 is the current stable release. Release notes are on the release page.

Compression is usually a bottleneck during backup operations, so pgBackRest solves this problem with parallel processing and more efficient compression algorithms like lz4 and zstd.

A custom protocol allows pgBackRest to backup, restore, and archive backups locally or remotely via TLS/SSH with minimal configuration. An interface to query PostgreSQL is also provided through the protocol layer so that remote access to PostgreSQL is never required, which increases security.

Multiple repositories allow, for example, a local repository with minimal retention for fast restores and a remote repository with longer retention for redundancy and accessibility across the enterprise.

Full, differential and incremental backups are supported. pgBackRest is not sensitive to rsync’s timing resolution issues, making differential and incremental backups safe without requiring a checksum for each file. Block-level backups save space by copying only the parts of files that have changed.

Retention policies for full and differential backups can be set to create coverage for any time frame. WAL archives can be maintained for all backups or strictly for the most recent backup. In the latter case the WAL required to make the old backup consistent will be retained in the archive.

The checksum is calculated for each file in the backup and rechecked during restore or verification. After copying the backup files, it waits until each WAL segment needed to make the backup consistent has reached the repository.

Backups in the repository can be stored in the same format as a standard PostgreSQL cluster (including tablespaces). If compression is disabled and hard links are enabled it is possible to do a backup snapshot in the repository and bring up the PostgreSQL cluster directly on the snapshot. This is beneficial for terabyte-scale databases that are time-consuming to restore using traditional methods.

All operations use file and directory level fsync to ensure durability.

If page checksum is enabled then pgBackRest will validate the checksum for each file copied during the backup. All page checksums are validated during full backups and checksums of files that have changed are validated during differential and incremental backups.

Validation failures do not stop the backup process, but warnings with precise details about which pages failed validation are output to the console and file logs.

This feature allows early detection of page-level corruption, before backups containing valid copies of the data are exhausted.

An interrupted backup can be resumed from the point where it was stopped. Already copied files are compared with the checksum in the manifest to ensure integrity. Since this operation can occur entirely on the repository host, it reduces the load on the PostgreSQL host and saves time because the checksum calculation is faster than compressing and retransmitting the data.

Compression and checksum calculations are performed in the stream when copying files to the repository, whether the repository is located locally or remotely.

If the repository is on the repository host, the compression is done on the PostgreSQL host and the files are transmitted in a compressed format and simply stored on the repository host. When compression is disabled a lower level of compression is used to make efficient use of the available bandwidth while keeping CPU costs to a minimum.

The manifest includes checksums for each file in the backup so that it is possible to use these checksums to greatly speed up processing during restore. On a delta restore any files that are not present in the backup are first removed and then a checksum is generated for the remaining files. The files that match the backup are left in place and the rest are restored as usual. Parallel processing can dramatically reduce restore times.

Dedicated commands for pushing WAL to the archive and getting WAL from the archive are included. Both commands support parallelism to speed up processing and run asynchronously to provide PostgreSQL the fastest possible response time.

Wall Push automatically detects wall segments that are pushed multiple times and de-duplicates if the segments are identical, otherwise an error is generated. Asynchronous wall push allows the transfer to be offloaded to another process that compresses wall segments in parallel for maximum throughput. This can be an important feature for databases with extremely high write volumes.

The asynchronous wall gate maintains a local queue of wall segments that are decompressed and ready for replay. This reduces the time it takes to provide the WAL to PostgreSQL thereby maximizing replay speed. High-latency connections and storage (like S3) benefit the most.

Both the push and get commands ensure that the database and repository match by comparing PostgreSQL versions and system identifiers. This virtually eliminates the possibility of misconfiguring the WAL storage location.

Tablespaces are fully supported and can be remapped to any location upon restore. It is also possible to remap all tablespaces to one location with a single command which is useful for development restores.

File and directory links are supported for any file or directory in the PostgreSQL cluster. When restoring it is possible to restore all links to their original locations, remap some or all links, or restore some or all links within the cluster directory as normal files or directories.

pgBackRest repositories can be located in S3, Azure, and GCS compatible object stores to allow virtually unlimited capacity and retention.

pgBackRest can encrypt repositories to secure backups, no matter where they are stored.

pgBackRest includes support for ten versions of PostgreSQL, five supported versions and the last five EOL versions. This allows enough time to upgrade to a supported version.

pgBackRest strives to make it easy to configure and operate:

pgBackRest would not exist without sponsors. Writing new features, fixing bugs, reviewing contributions, answering community questions, and maintenance all take a lot of time.

Current sponsor: Supabase.

Previous Sponsors: Crunchy Data, Resonate.

Armchair graphic by Alexander Skowalski.