Professional Documents
Culture Documents
PerconaXtrabackup-8 0
PerconaXtrabackup-8 0
Documentation
8.0.32-26 (April 4, 2023)
Table of contents
2. Introduction 7
3. Installation 12
4. Run in Docker 25
6. Backup scenarios 33
7. User’s manual 47
8. Advanced features 48
9. Security 72
11.9 Use the xbcloud binary with Microsoft Azure Cloud Storage 106
14.1 Error Message: Found tables with row versions due to INSTANT ADD/DROP columns 142
This documentation is for the latest release: Percona XtraBackup 8.0.32-26 (Release Notes).
Percona XtraBackup is an open-source hot backup utility, for MySQL - based servers, that does not lock your
database during the backup.
Percona XtraBackup can back up data from InnoDB, XtraDB, MyISAM, and MyRocks tables on MySQL 8.0
servers as well as Percona Server for MySQL with XtraDB, Percona Server for MySQL 8.0, and Percona XtraDB
Cluster 8.0.
Version updates
Version 8.0.6 and later supports the MyRocks storage engine. An incremental backup on the MyRocks storage
engine does not determine if an earlier full or incremental backup contains the same files. Percona XtraBackup
copies all MyRocks files each time it takes a backup. Percona XtraBackup does not support the TokuDB storage
engine.
See also
Percona TokuBackup
Percona XtraBackup 8.0 does not support making backups of databases created in versions prior to 8.0 of
MySQL, Percona Server for MySQL or Percona XtraDB Cluster. As the changes that MySQL 8.0 introduced in
data dictionaries, redo log and undo log are incompatible with previous versions, it is currently impossible
for Percona XtraBackup 8.0 to also support versions prior to 8.0.
Due to changes in MySQL 8.0.20 released by Oracle at the end of April 2020, Percona XtraBackup 8.0, up to
version 8.0.11, is not compatible with MySQL version 8.0.20 or higher, or Percona products that are based on it:
Percona Server for MySQL and Percona XtraDB Cluster.
For more information, see Percona XtraBackup 8.x and MySQL 8.0.20
For a high-level overview of many of its advanced features, including a feature comparison, please see
About Percona XtraBackup.
1.0.1 Contact us
For paid support and managed or consulting services , contact Percona Sales.
2. Introduction
Percona XtraBackup makes MySQL hot backups for all versions of Percona Server for MySQL, and MySQL. It
performs streaming, compressed, and incremental MySQL backups.
Important
Percona XtraBackup 2.4 supports MySQL and Percona Server for MySQL 5.6 and 5.7 databases. Due to changes in
the MySQL redo log and data dictionary formats, the Percona XtraBackup 8.0.x versions are only compatible with
MySQL 8.0.x, Percona Server for MySQL 8.0.x, and compatible versions.
Percona’s enterprise-grade commercial MySQL Support contracts include support for Percona XtraBackup.
We recommend support for critical production deployments. Percona XtraDB Backup supports encryption.
Percona XtraBackup works with MySQL and Percona Server. It supports completely non-blocking backups of
InnoDB, XtraDB, and MyRocks storage engines. Fast incremental backups are supported for Percona Server
with the XtraDB changed page tracking enabled.
In addition, it can back up the following storage engines by briefly pausing writes at the end of the backup:
MyISAM and Merge, including partitioned tables, triggers, and database options. The InnoDB tables are still
locked while copying non-InnoDB data.
Version updates
Version 8.0.6 and later supports the MyRocks storage engine. An incremental backup on the MyRocks storage
engine does not determine if an earlier full or incremental backup contains the same files. Percona XtraBackup
copies all MyRocks files each time it takes a backup. Percona XtraBackup does not support the TokuDB storage
engine.
See also
Percona TokuBackup
Here is a short list of the Percona XtraBackup features. See the documentation for more.
• Percona XtraBackup skips secondary index pages and recreates them when a compact backup is
prepared
• Percona XtraBackup can export individual tables even from a full backup, regardless of the InnoDB
version
• Backup locks is a lightweight alternative to FLUSH TABLES WITH READ LOCK available in Percona Server.
Percona XtraBackup uses them automatically to copy non-InnoDB data to avoid blocking DML queries
that modify InnoDB tables.
See also
Additional information
Contact us
For paid support and managed or consulting services , contact Percona Sales.
This works because InnoDB maintains a redo log, also called the transaction log. This contains a record of
every change to InnoDB data. When InnoDB starts, it inspects the data files and the transaction log, and
performs two steps. It applies committed transaction log entries to the data files, and it performs an undo
operation on any transactions that modified data but did not commit.
Percona XtraBackup works by remembering the LSN when it starts, and then copying away the data files. It
takes some time to do this, so if the files are changing, then they reflect the state of the database at
different points in time. At the same time, Percona XtraBackup runs a background process that watches the
transaction log files, and copies changes from it. Percona XtraBackup needs to do this continually because
the transaction logs are written in a round-robin fashion, and can be reused after a while. Percona
XtraBackup needs the transaction log records for every change to the data files since it began execution.
Percona XtraBackup uses Backup locks where available as a lightweight alternative to FLUSH TABLES WITH
READ
LOCK . This feature is available in Percona Server for MySQL 5.6+. MySQL 8.0 allows acquiring an instance level
backup lock via the LOCK INSTANCE FOR BACKUP statement.
Locking is only done for MyISAM and other non-InnoDB tables after Percona XtraBackup finishes backing up
all InnoDB/XtraDB data and logs. Percona XtraBackup uses this automatically to copy non-InnoDB data to
avoid blocking DML queries that modify InnoDB tables.
Important
The BACKUP_ADMIN privilege is required to query the performance_schema_log_status for either LOCK
INSTANCE FOR BACKUP or LOCK TABLES FOR BACKUP .
xtrabackup tries to avoid backup locks and FLUSH TABLES WITH READ LOCK when the instance contains only
InnoDB tables. In this case, xtrabackup obtains binary log coordinates from performance_schema.log_status .
FLUSH
TABLES WITH READ LOCK is still required in MySQL 8.0 when xtrabackup is started with the --slave-info . The
log_status table in Percona Server for MySQL 8.0 is extended to include the relay log coordinates, so no
locks are needed even with the --slave-info option.
See also
When backup locks are supported by the server, xtrabackup first copies InnoDB data, runs the LOCK TABLES
FOR BACKUP and then copies the MyISAM tables. Once this is done, the backup of the files will begin. It will
backup .frm, .MRG, .MYD, .MYI, .CSM, .CSV, .sdi and .par files.
After that xtrabackup will use LOCK BINLOG FOR BACKUP to block all operations that might change either binary
log position or Exec_Master_Log_Pos or Exec_Gtid_Set (i.e. source binary log coordinates corresponding to
the current SQL thread state on a replication replica) as reported by SHOW MASTER/SLAVE STATUS . xtrabackup
will then finish copying the REDO log files and fetch the binary log coordinates. After this is completed
xtrabackup will unlock the binary log and tables.
Finally, the binary log position will be printed to STDERR and xtrabackup will exit returning 0 if all went OK.
Note that the STDERR of xtrabackup is not written in any file. You will have to redirect it to a file, e.g.,
xtrabackup OPTIONS 2> backupout.log .
It will also create the following files in the directory of the backup.
During the prepare phase, Percona XtraBackup performs crash recovery against the copied data files, using
the copied transaction log file. After this is done, the database is ready to restore and use.
The backed-up MyISAM and InnoDB tables will be eventually consistent with each other, because after the
prepare (recovery) process, InnoDB’s data is rolled forward to the point at which the backup completed, not
rolled back to the point at which it started. This point in time matches where the FLUSH
TABLES WITH READ LOCK was taken, so the MyISAM data and the prepared InnoDB data are in sync.
The xtrabackup offers many features not mentioned in the preceding explanation. The functionality of each
tool is explained in more detail further in this manual. In brief, though, the tools enable you to do operations
such as streaming and incremental backups with various combinations of copying the data files, copying
the log files, and applying the logs to the data.
To restore a backup with xtrabackup you can use the --copy-back or --move-back options.
xtrabackup will read from the my.cnf the variables datadir, innodb_data_home_dir,
innodb_data_file_path, innodb_log_group_home_dir and check that the directories exist.
It will copy the MyISAM tables, indexes, etc. (.MRG, .MYD, .MYI, .CSM, .CSV, .sdi , and par files) first, InnoDB
tables and indexes next and the log files at last. It will preserve file’s attributes when copying them, you may
have to change the files’ ownership to mysql before starting the database server, as they will be owned by
the user who created the backup.
Alternatively, the --move-back option may be used to restore a backup. This option is similar to --copy-back
with the only difference that instead of copying files it moves them to their target locations. As this option
removes backup files, it must be used with caution. It is useful in cases when there is not enough free disk
space to hold both data files and their backup copies.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
8.0.30 -23
Percona uses semantic version numbering, which follows the pattern of base version and build version.
Percona assigns unique, non-negative integers in increasing order for each version release. The version
number combines the base MySQL 8.0 version number and the minor build version.
Note that Percona XtraBackup numbering changed after the 8.0.14 version to align Percona XtraBackup
versions with Percona Server and MySQL. Find more information in Aligning Percona XtraBackup Versions
with Percona Server for MySQL blog post.
The version numbers for Percona XtraBackup 8.0.30-23 define the following information:
• Base version - the leftmost numbers indicate the MySQL 8.0 version used as a base. An increase in base
version resets the minor build version to 0.
• Minor build version - an internal number that denotes the version of the software. A build version
increases by one each time the Percona XtraBackup is released.
Percona XtraBackup 8.0.28-20 and 8.0.28-21 are multiple releases based on MySQL 8.0.28.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
3. Installation
• Install Percona XtraBackup from a Binary Tarball with all the required files and binaries for a manual
installation
Before installing Percona XtraBackup with any of these methods, we recommend that you review the release
notes
Contact us
For paid support and managed or consulting services , contact Percona Sales.
Specific information on the supported platforms, products, and versions is described in Percona Release
Lifecycle Overview.
To check what data each DEB package contains, see What’s in the packages.
Important
To prevent intermittent backup failures, update the curl utility in Debian 10.
Percona XtraBackup, like many other Percona products, is installed with the percona-release package
configuration tool.
1. Download a DEB package for percona-release the repository packages from Percona web:
{.bash data-prompt="$"}
$ wget https://repo.percona.com/apt/percona-release_latest.$(lsb_release -sc)_all.deb
2. Install the downloaded package with dpkg. To do that, run the following commands as root or with sudo:
dpkg -i percona-release_latest.$(lsb_release -sc)_all.deb
Once you install this package the Percona repositories should be added. You can check the repository
setup in the /etc/apt/sources.list.d/percona-release.list file.
If Percona XtraBackup is intended to be used in combination with the upstream MySQL Server, you only
need to enable the tools repository: percona-release enable-only tools .
4. Remember to update the local cache: apt update
{.bash data-prompt="$"}
$ sudo apt install percona-xtrabackup-80
6. To decompress backups made using LZ4 or ZSTD compression algorithm, install the corresponding
package:
{.bash data-prompt="$"}
$ sudo apt install lz4
{.bash data-prompt="$"}
$ sudo apt install zstd
Note
In some cases you might need to “pin” the selected packages to avoid the upgrades from the distribution
repositories. Make a new file /etc/apt/preferences.d/00percona.pref and add the following lines in it:
Package: *
Pin: release o=Percona Development Team
Pin-Priority: 1001
For more information about the pinning, check the official debian wiki.
See also
To install Percona XtraBackup using downloaded deb packages, see Install with package manager.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
The easiest way to install the Percona Yum repository is to install an RPM that configures yum and installs
the Percona GPG key.
Specific information on the supported platforms, products, and versions is described in Percona Software
and Platform Lifecycle.
To check what data each RPM package contains, see What’s in the packages.
1. Install the Percona yum repository by running the following command as the root user or with sudo:
{.bash data-prompt="$"}
$ sudo yum install \
https://repo.percona.com/yum/percona-release-latest.\
noarch.rpm
{.bash data-prompt="$"}
$ sudo percona-release enable-only tools release
If Percona XtraBackup is intended to be used in combination with the upstream MySQL Server, you only
need to enable the `tools repository:
{.bash data-prompt="$"}
$ sudo percona-release enable-only tools
{.bash data-prompt="$"}
$ sudo yum install percona-xtrabackup-80
Warning
Make sure that you have the libev package installed before installing Percona XtraBackup on CentOS 6. For this
operating system, the libev package is available from the EPEL repositories.
4. To decompress backups made using LZ4 or ZSTD compression algorithm, install the corresponding
package:
{.bash data-prompt="$"}
$ sudo yum install lz4
{.bash data-prompt="$"}
$ sudo yum install zstd
See also
To install Percona XtraBackup using downloaded rpm packages, see Install with package manager.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
On Debian / Ubuntu
Download DEB packages of the desired series for your architecture from Download Percona XtraBackup 8.0.
The following example downloads Percona XtraBackup 8.0.26-18 release package for Ubuntu 20.04:
{.bash data-prompt="$"}
$ wget https://downloads.percona.com/downloads/Percona-XtraBackup-LATEST/Percona-XtraBackup-8.0.26-18/
binary/debian/focal/x86_64/percona-xtrabackup-80_8.0.26-18-1.focal_amd64.deb
{.bash data-prompt="$"}
$ sudo dpkg -i percona-xtrabackup-80_8.0.26-18-1.focal_amd64.deb
When installing packages manually like this, resolve all the dependencies and install missing packages
yourself.
Download RPM packages of the desired series for your architecture from the download page. The following
example downloads Percona XtraBackup 8.0.4 release package for CentOS 7:
{.bash data-prompt="$"}
$ wget https://www.percona.com/downloads/XtraBackup/Percona-XtraBackup-8.0.4/binary/redhat/7/x86_64/
percona-xtrabackup-80-8.0.4-1.el7.x86_64.rpm
{.bash data-prompt="$"}
$ yum localinstall percona-xtrabackup-80-8.0.4-1.el7.x86_64.rpm
When installing packages manually like this, you’ll need to make sure to resolve all the dependencies and
install missing packages yourself.
See also
Contact us
For paid support and managed or consulting services , contact Percona Sales.
The following table lists the tarball types available in Linux - Generic . Select the Percona XtraBackup 8.0
version, the software or the operating system, and the type of tarball for your installation. Binary tarballs
support all distributions.
After you have downloaded the binary tarballs, extract the tarball in the file location of your choice.
Select a different software, such as Ubuntu 20.04 (Focal Fossa), for a tarball for that operating system. You
can download the packages together or separately.
The following command is an example of downloading the full tarball for Linux/Generic:
$ wget https://downloads.percona.com/downloads/Percona-XtraBackup-LATEST/Percona-
XtraBackup-8.0.23-16/binary/tarball/percona-xtrabackup-8.0.23-16-Linux-
x86_64.glibc2.17.tar.gz
Contact us
For paid support and managed or consulting services , contact Percona Sales.
The following packages and tools must be installed to compile Percona XtraBackup from source. These
might vary from system to system.
Important
To build **Percona XtraBackup 8.0 from source, you must use cmake version 3. To check which version is currently
installed, run cmake --version at a command prompt. If the version is not 3 , install cmake3 .
This cmake version may be available in your distribution as a separate package cmake3 . For more
information, see cmake.org
shell
sudo apt install bison pkg-config cmake devscripts debconf \
debhelper automake bison ca-certificates libprocps-dev \
libcurl4-openssl-dev cmake debhelper libaio-dev \
libncurses-dev libssl-dev libtool libz-dev libgcrypt-dev libev-dev libprocps-dev \
lsb-release build-essential rsync libdbd-mysql-perl \
libnuma1 socat librtmp-dev libtinfo5 vim-common \
liblz4-tool liblz4-1 liblz4-dev zstd python-docutils qpress
{.bash data-prompt="$"}
$ sudo apt install python3-sphinx
Percona Xtrabackup requires GCC version 5.3 or higher. If the version of GCC installed on your system is
lower then you may need to install and enable the Developer Toolset on RPM -based distributions to make
sure that you use the latest GCC compiler and development tools. Then, install cmake and other
dependencies:
{.bash data-prompt="$"}
$ sudo yum install cmake openssl-devel libaio libaio-devel automake autoconf \
bison libtool ncurses-devel libgcrypt-devel libev-devel libcurl-devel zlib-devel \
zstd vim-common procps-ng-devel
{.bash data-prompt="$"}
$ sudo yum install python3-sphinx
At this step, you have cmake run the commands in the CMakeList.txt file to generate the build pipeline, i.e. a
native build environment that will be used to compile the source code).
1. Change to the directory where you cloned the Percona XtraBackup repository
{.bash data-prompt="$"}
$ cd percona-xtrabackup
2. Create a directory to store the compiled files and then change to that directory:
{.bash data-prompt="$"}
$ mkdir build
$ cd build
3. Run cmake or cmake3. In either case, the options you need to use are the same.
Note
You can build Percona XtraBackup with man pages but this requires python-sphinx package which isn’t available
from that main repositories for every distribution. If you installed the python-sphinx package you need to remove
the -DWITH_MAN_PAGES=OFF from previous command.
Parameter Information
Parameter Description
-DWITH_BOOST For the -DWITH_BOOST parameter, specify the name of a directory to download the
boost library to. This directory is created automatically in your current directory.
- To build Percona XtraBackup man pages, use ON or remove this parameter from the
DWITH_MAN_PAGES command line (it is ON by default). To install the man pages, install the python3-
sphinx package first. See also ## Step 2: Compiling the source code
-B (--build) Percona XtraBackup is configured to forbid generating the build pipeline for make in
the same directory where you store your sources. The -B parameter refers to the
directory that contains the source code. In this example, we use the relative path to
the parent directory (..).
Important
CMake Error at CMakeLists.txt:367 (MESSAGE): Please do not build in-source. Out-of source builds are highly
recommended: you can have multiple builds for the same source, and there is an easy way to do cleanup, simply
remove the build directory (note that ‘make clean’ or ‘make distclean’ does not work)
To compile the source code in your build directory, use the make command.
1. Change to the build directory (created at Step 2: Generating the build pipeline).
2. Run the make command. This command may take a long time to complete.
{.bash data-prompt="$"}
$ make To use all CPU threads and make compilation faster please use:
{.bash data-prompt="$"}
$ make -j$(nproc --all)
The following command installs all Percona XtraBackup binaries xtrabackup and tests to default location on
the target system: /usr/local/xtrabackup .
You may use the DESTDIR parameter with make install to install Percona XtraBackup to another location.
Make sure that the effective user is able to write to the destination you choose.
In fact, the destination directory is determined by the installation layout ( -DINSTALL_LAYOUT ) that cmake
applies (see Step 2: Generating the build pipeline). In addition to the installation directory, this parameter
controls a number of other destinations that you can adjust for your system.
By default, this parameter is set to STANDALONE , which implies the installation directory to be /usr/local/
xtrabackup .
See also
After Percona XtraBackup is installed on your system, you may run it by using the full path to the xtrabackup
command:
$ /usr/local/xtrabackup/bin/xtrabackup
Update your PATH environment variable if you would like to use the command on the command line directly.
Alternatively, you may consider placing a soft link (using ln -s ) to one of the locations listed in your PATH
environment variable.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
DEB packages
Package Contains
RPM packages
Package Contains
Contact us
For paid support and managed or consulting services , contact Percona Sales.
{.bash data-prompt="$"}
$ sudo apt remove percona-xtrabackup-80
{.bash data-prompt="$"}
$ yum remove percona-xtrabackup
Contact us
For paid support and managed or consulting services , contact Percona Sales.
4. Run in Docker
You can run Percona XtraBackup in a Docker container without installing the product. All required libraries
are available in the container. Being a lightweight execution environment, Docker containers enable
creating configurations where each program runs in a separate container. You may run Percona Server for
MySQL in one container and Percona XtraBackup in another. Docker images offer a range of options.
Create a Docker container based on a Docker image. Docker images for Percona XtraBackup are hosted
publicly on Docker Hub.
This section demonstrates how to back up data on a Percona Server for MySQL running in another Dockers
container.
Your operating system may already provide a package for docker. However, the versions of Docker provided
by your operating system are likely to be outdated.
Use the installation instructions for your operating system available from the Docker site to set up the latest
version of docker.
Percona XtraBackup works in combination with a database server. When running a Docker container for
Percona XtraBackup, you can make backups for a database server either installed on the host machine or
running in a separate Docker container.
To set up a database server on a host machine or in Docker container, follow the documentation of the
supported product that you intend to use with Percona XtraBackup.
See also
As soon as Percona Server for MySQL runs, add some data to it. Now, you are ready to make backups with
Percona XtraBackup.
Important
When running Percona XtraBackup from a container and connecting to a MySQLserver container, we recommend
using the –user root option in the Docker command.
You can create a Docker container based on Percona XtraBackup image with either docker create or the
docker run command. docker create creates a Docker container and makes it available for starting later.
Docker downloads the Percona XtraBackup image from the Docker Hub. If it is not the first time you use the
selected image, Docker uses the image available locally.
With parameter name you give a meaningful name to your new Docker container so that you could easily
locate it among your other containers.
The volumes-from flag refers to Percona Server for MySQL and indicates that you intend to use the same
data as the Percona Server for MySQL container.
Run the container with exactly the same parameters that were used when the container was created:
This command starts the percona-xtrabackup container, attaches to its input/output streams, and opens
an interactive shell.
The docker run is a shortcut command that creates a Docker container and then immediately runs it.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
xtrabackup opens the source data files in read-write mode, although it does not modify the files. This means
that you must run xtrabackup as a user who has permission to write the data files. The reason for opening
the files in read-write mode is that xtrabackup uses the embedded InnoDB libraries to open and read the
files, and InnoDB opens them in read-write mode because it normally assumes it is going to write to them.
Because xtrabackup reads large amounts of data from the filesystem, it uses posix_fadvise() where
possible, to instruct the operating system not to try to cache the blocks it reads from disk. Without this hint,
the operating system would prefer to cache the blocks, assuming that xtrabackup is likely to need them
again, which is not the case. Caching such large files can place pressure on the operating system’s virtual
memory and cause other processes, such as the database server, to be swapped out. The xtrabackup tool
avoids this with the following hint on both the source and destination files:
posix_fadvise(file, 0, 0, POSIX_FADV_DONTNEED)
In addition, xtrabackup asks the operating system to perform more aggressive read-ahead optimizations
on the source files:
posix_fadvise(file, 0, 0, POSIX_FADV_SEQUENTIAL)
When copying the data files to the target directory, xtrabackup reads and writes 1 MB of data at a time. This
is not configurable. When copying the log file, xtrabackup reads and writes 512 bytes at a time. This is also
not possible to configure, and matches InnoDB’s behavior (workaround exists in Percona Server for MySQL
because it has an option to tune innodb_log_block_size for XtraDB, and in that case Percona XtraBackup will
match the tuning).
After reading from the files, xtrabackup iterates over the 1MB buffer a page at a time, and checks for page
corruption on each page with InnoDB’s buf_page_is_corrupted() function. If the page is corrupt, it re-reads
and retries up to 10 times for each page. It skips this check on the doublewrite buffer.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
Privilege refers to the operations that a system user is permitted to do in the database server. They are set
at the database server and only apply to users in the database server.
Permissions are those which permits a user to perform operations on the system, like reading, writing or
executing on a certain directory or start/stop a system service. They are set at a system level and only
apply to system users.
When xtrabackup is used, there are two actors involved: the user invoking the program - a system user -
and the user performing action in the database server - a database user. Note that these are different users
in different places, even though they may have the same username.
All the invocations of xtrabackup in this documentation assume that the system user has the appropriate
permissions, and you are providing the relevant options for connecting the database server - besides the
options for the action to be performed - and the database user has adequate privileges.
The database user used to connect to the server and its password are specified by the --user and --
password option:
If you don’t use the --user option, Percona XtraBackup will assume the database user whose name is the
system user executing it.
According to your system, you may need to specify one or more of the following options to connect to the
server:
Option Description
-port Use this port when connecting to the database with TCP/IP
-host Use this host when connecting to the database server with TCP/IP
These options are passed to the mysql child process without alteration, see mysql --help for details.
Note
In case of multiple server instances, the correct connection parameters (port, socket, host) must be specified in
order for xtrabackup to talk to the correct server.
Once connected to the server, in order to perform a backup you will need READ and EXECUTE permissions at
a filesystem level in the server’s datadir.
The database user needs the following privileges on the tables or databases to be backed up:
• RELOAD and LOCK TABLES (unless the --no-lock option is specified) in order to run FLUSH TABLES WITH
READ LOCK and FLUSH ENGINE LOGS prior to start copying the files, and requires this privilege when
Backup Locks are used
• BACKUP_ADMIN privilege is needed to query the performance_schema.log_status table, and run LOCK
INSTANCE FOR BACKUP , LOCK BINLOG FOR BACKUP , or LOCK TABLES FOR BACKUP .
• PROCESS in order to run SHOW ENGINE INNODB STATUS (which is mandatory), and optionally to see all
threads which are running on the server (see FLUSH TABLES WITH READ LOCK option),
• SUPER in order to start/stop the replication threads in a replication environment, use XtraDB Changed
Page Tracking for Incremental Backups and for handling FLUSH TABLES WITH READ LOCK,
• SELECT privilege on the replication_group_members table to validate if the instance is part of group
replication cluster.
The explanation of when these are used can be found in How Percona XtraBackup Works.
An SQL example of creating a database user with the minimum privileges required to full backups would be:
Contact us
For paid support and managed or consulting services , contact Percona Sales.
The xtrabackup binary reads the [mysqld] and [xtrabackup] sections from any configuration files, in that
order. That is so that it can read its options from your existing MySQL installation, such as the datadir or
some of the InnoDB options. If you want to override these, just specify them in the [xtrabackup] section, and
because it is read later, it will take precedence.
You don’t need to put any configuration in your my.cnf if you don’t want to. You can simply specify the
options on the command-line. Normally, the only thing you might find convenient to place in the
[xtrabackup] section of your my.cnf file is the target_dir option to default the directory in which the
backups will be placed, for example:
[xtrabackup]
target_dir = /data/backups/mysql/
This manual will assume that you do not have any file-based configuration for xtrabackup, so it will always
show command-line options being used explicitly. Please see the option and variable reference for details
on all the configuration options.
The xtrabackup binary does not accept exactly the same syntax in the my.cnf file as the mysqld server
binary does. For historical reasons, the mysqld server binary accepts parameters with a --set-
variable=<variable>=<value> syntax, which xtrabackup does not understand. If your my.cnf file has such
configuration directives, you should rewrite them in the --variable=value syntax.
The xtrabackup tool requires no special configuration on most systems. However, the storage where the --
target-dir is located must behave properly when fsync() is called. In particular, we have noticed that if
you mount the NFS volume without the sync option the NFS volume does not sync the data. As a result, if you
back up to an NFS volume mounted with the async option, and then try to prepare the backup from a
different server that also mounts that volume, the data might appear to be corrupt. You can use the sync
mount option to avoid this problem.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
See also
Percona XtraBackup 8.0.21 adds the --no-server-version-check option. Before the backup starts, XtraBackup
compares the source system version to the Percona XtraBackup version. If the source system version is
greater than the XtraBackup version, XtraBackup stops the backup and returns an error message. This
comparison prevents a failed backup or a corrupted backup due to source system changes.
• The source system and the PXB version are the same, the backup proceeds
• The source system is less than the PXB version, the backup proceeds
• The source system is greater than the PXB version, and the parameter is not overridden, the backup is
stopped and returns an error message
• The source system is greater than the PXB version, and the parameter is overridden, the backup
proceeds
Explicitly adding the --no-server-version-check parameter, like the example, overrides the parameter and
the backup proceeds.
When you override the parameter, the following events can happen:
• Backup fails
Contact us
For paid support and managed or consulting services , contact Percona Sales.
In certain cases, the exit value can be something other than 0 or 1, due to the command-line option code
included from the MySQL libraries. An unknown command-line option, for example, will cause an exit code of
255.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
6. Backup scenarios
To create a backup, run xtrabackup with the --backup option. You also need to specify the --target-dir
option, which is where the backup will be stored, if the InnoDB data or log files are not stored in the same
directory, you might need to specify the location of those, too. If the target directory does not exist,
xtrabackup creates it. If the directory does exist and is empty, xtrabackup will succeed.
xtrabackup will not overwrite existing files, it will fail with operating system error 17, file exists .
This will store the backup at /data/backups/ . If you specify a relative path, the target directory will be relative
to the current directory.
During the backup process, you should see a lot of output showing the data files being copied, as well as the
log file thread repeatedly scanning the log files and copying from it. Here is an example that shows the log
thread scanning the log in the background, and a file copying thread working on the ibdata1 file:
Expected output
{.text .no-copy}
160906 10:19:17 Finished backing up non-InnoDB tables and files
160906 10:19:17 Executing FLUSH NO_WRITE_TO_BINLOG ENGINE LOGS...
xtrabackup: The latest check point (for incremental): '62988944'
xtrabackup: Stopping log copying thread.
.160906 10:19:18 >> log scanned up to (137343534)
160906 10:19:18 Executing UNLOCK TABLES
160906 10:19:18 All tables unlocked
160906 10:19:18 Backup created in directory '/data/backups/'
160906 10:19:18 [00] Writing backup-my.cnf
160906 10:19:18 [00] ...done
160906 10:19:18 [00] Writing xtrabackup_info
160906 10:19:18 [00] ...done
xtrabackup: Transaction log of lsn (26970807) to (137343534) was copied.
160906 10:19:18 completed OK!
The last thing you should see is something like the following, where the value of the <LSN> will be a number
that depends on your system:
Note
Log copying thread checks the transactional log every second to see if there were any new log records written
that need to be copied, but there is a chance that the log copying thread might not be able to keep up with the
amount of writes that go to the transactional logs, and will hit an error when the log records are overwritten before
they could be read.
After the backup is finished, the target directory will contain files such as the following, assuming you have a
single InnoDB table test.tbl1 and you are using MySQL’s innodb_file_per_table option:
$ ls -lh /data/backups/
Expected output
{.text .no-copy}
total 182M
drwx------ 7 root root 4.0K Sep 6 10:19 .
drwxrwxrwt 11 root root 4.0K Sep 6 11:05 ..
-rw-r----- 1 root root 387 Sep 6 10:19 backup-my.cnf
-rw-r----- 1 root root 76M Sep 6 10:19 ibdata1
drwx------ 2 root root 4.0K Sep 6 10:19 mysql
drwx------ 2 root root 4.0K Sep 6 10:19 performance_schema
drwx------ 2 root root 4.0K Sep 6 10:19 sbtest
drwx------ 2 root root 4.0K Sep 6 10:19 test
drwx------ 2 root root 4.0K Sep 6 10:19 world2
-rw-r----- 1 root root 116 Sep 6 10:19 xtrabackup_checkpoints
-rw-r----- 1 root root 433 Sep 6 10:19 xtrabackup_info
-rw-r----- 1 root root 106M Sep 6 10:19 xtrabackup_logfile
The backup can take a long time, depending on how large the database is. It is safe to cancel at any time,
because xtrabackup does not modify the database.
After making a backup with the --backup option, you need to prepare it in order to restore it. Data files are
not point-in-time consistent until they are prepared, because they were copied at different times as the
program ran, and they might have been changed while this was happening.
If you try to start InnoDB with these data files, it will detect corruption and stop working to avoid running on
damaged data. The --prepare step makes the files perfectly consistent at a single instant in time, so you
can run InnoDB on them.
You can run the prepare operation on any machine; it does not need to be on the originating server or the
server to which you intend to restore. You can copy the backup to a utility server and prepare it there.
Note that Percona XtraBackup 8.0 can only prepare backups of MySQL 8.0, Percona Server for MySQL 8.0,
and Percona XtraDB Cluster 8.0 databases. Releases prior to 8.0 are not supported.
During the prepare operation, xtrabackup boots up a kind of modified embedded InnoDB (the libraries
xtrabackup was linked against). The modifications are necessary to disable InnoDB standard safety checks,
such as complaining about the log file not being the right size. This warning is not appropriate for working
with backups. These modifications are only for the xtrabackup binary; you do not need a modified InnoDB to
use xtrabackup for your backups.
The prepare step uses this “embedded InnoDB” to perform crash recovery on the copied data files, using the
copied log file. The prepare step is very simple to use: you simply run xtrabackup with the --prepare option
and tell it which directory to prepare, for example, to prepare the previously taken backup run:
When this finishes, you should see an InnoDB shutdown with a message such as the following, where again
the value of LSN will depend on your system:
Expected output
{.text .no-copy}
InnoDB: Shutdown completed; log sequence number 137345046
160906 11:21:01 completed OK!
All following prepares will not change the already prepared data files, you’ll see that output says:
Expected output
{.text .no-copy}
xtrabackup: This target seems to be already prepared.
xtrabackup: notice: xtrabackup_logfile was already used to '--prepare'.
It is not recommended to interrupt xtrabackup process while preparing backup because it may cause data
files corruption and backup will become unusable. Backup validity is not guaranteed if prepare process was
interrupted.
Note
If you intend the backup to be the basis for further incremental backups, you should use the --apply-log-only
option when preparing the backup, or you will not be able to apply incremental backups to it. See the
documentation on preparing incremental backups for more details.
Warning
For convenience, xtrabackup binary has the --copy-back option to copy the backup to the datadir of the
server:
If you don’t want to save your backup, you can use the --move-back option which will move the backed up
data to the datadir.
If you don’t want to use any of the above options, you can additionally use rsync or cp to restore the files.
Note
The datadir must be empty before restoring the backup. Also, it’s important to note that MySQL server needs to be
shut down before restore is performed. You cannot restore to a datadir of a running mysqld instance (except
when importing a partial backup).
Example of the rsync command that can be used to restore the backup can look like this:
You should check that the restored files have the correct ownership and permissions.
As files’ attributes will be preserved, in most cases you will need to change the files’ ownership to mysql
before starting the database server, as they will be owned by the user who created the backup:
Contact us
For paid support and managed or consulting services , contact Percona Sales.
Note
Incremental backups on the MyRocks storage engine do not determine if an earlier full backup or incremental
backup contains the same files. Percona XtraBackup copies all the MyRocks files each time it takes a backup.
You can perform many incremental backups between each full backup, so you can set up a backup
process such as a full backup once a week and an incremental backup every day, or full backups every day
and incremental backups every hour.
Incremental backups work because each InnoDB page contains a log sequence number, or LSN. The LSN is
the system version number for the entire database. Each page’s LSN shows how recently it was changed.
An incremental backup copies each page whose LSN is newer than the previous incremental or full backup’s
LSN. An algorithm finds the pages that match the criteria. The algorithm reads the data pages and checks
the page LSN.
Percona XtraBackup 8.0.30 removes the algorithm that used the changed page tracking feature in Percona
Server for MySQL. Percona Server for MySQL 8.0.27 deprecated the changed page tracking feature.
With PXB 8.0.27 or earlier, another algorithm enabled the Percona Server for MySQL changed page tracking
feature. The algorithm generates a bitmap file. The xtrabackup binary uses that bitmap file to read only
those pages needed for the incremental backup. This method potentially saves resources. The backup
enables the algorithm by default if the xtrabackup binary discovers the bitmap file. You can override the
algorithm with --incremental-force-scan which forces a read of all pages even if the bitmap file is available.
To make an incremental backup, begin with a full backup as usual. The xtrabackup binary writes a file called
xtrabackup_checkpoints into the backup’s target directory. This file contains a line showing the to_lsn , which
is the database’s LSN at the end of the backup. Create the full backup with a following command:
If you look at the xtrabackup_checkpoints file, you should see similar content depending on your LSN nuber:
Expected output
{.text .no-copy}
backup_type = full-backuped
from_lsn = 0
to_lsn = 1626007
last_lsn = 1626007
compact = 0
recover_binlog_info = 1
Now that you have a full backup, you can make an incremental backup based on it. Use the following
command:
The /data/backups/inc1/ directory should now contain delta files, such as ibdata1.delta and test/
table1.ibd.delta . These represent the changes since the LSN 1626007 . If you examine the
xtrabackup_checkpoints file in this directory, you should see similar content to the following:
Expected output
{.text .no-copy}
backup_type = incremental
from_lsn = 1626007
to_lsn = 4124244
last_lsn = 4124244
compact = 0
recover_binlog_info = 1
from_lsn is the starting LSN of the backup and for incremental it has to be the same as to_lsn (if it is the
last checkpoint) of the previous/base backup.
It’s now possible to use this directory as the base for yet another incremental backup:
Expected output
{.text .no-copy}
backup_type = incremental
from_lsn = 4124244
to_lsn = 6938371
last_lsn = 7110572
compact = 0
recover_binlog_info = 1
Note
In this case you can see that there is a difference between the to_lsn (last checkpoint LSN) and last_lsn (last
copied LSN), this means that there was some traffic on the server during the backup process.
The --prepare step for incremental backups is not the same as for full backups. In full backups, two types of
operations are performed to make the database consistent: committed transactions are replayed from the
log file against the data files, and uncommitted transactions are rolled back. You must skip the rollback of
uncommitted transactions when preparing an incremental backup, because transactions that were
uncommitted at the time of your backup may be in progress, and it’s likely that they will be committed in
the next incremental backup. You should use the --apply-log-only option to prevent the rollback phase.
Warning
If you do not use the --apply-log-only option to prevent the rollback phase, then your incremental backups will
be useless. After transactions have been rolled back, further incremental backups cannot be applied.
Beginning with the full backup you created, you can prepare it, and then apply the incremental differences
to it. Recall that you have the following backups:
/data/backups/base
/data/backups/inc1
/data/backups/inc2
To prepare the base backup, you need to run --prepare as usual, but prevent the rollback phase:
Expected output
{.text .no-copy}
InnoDB: Shutdown completed; log sequence number 1626007
161011 12:41:04 completed OK!
The log sequence number should match the to_lsn of the base backup, which you saw previously.
Warning
This backup is actually safe to restore as-is now, even though the rollback phase has been skipped. If you restore
it and start MySQL, InnoDB will detect that the rollback phase was not performed, and it will do that in the
background, as it usually does for a crash recovery upon start. It will notify you that the database was not shut
down normally.
To apply the first incremental backup to the full backup, run the following command:
This applies the delta files to the files in /data/backups/base , which rolls them forward in time to the time of
the incremental backup. It then applies the redo log as usual to the result. The final data is in /data/backups/
base , not in the incremental directory. You should see an output similar to:
Expected output
{.text .no-copy}
incremental backup from 1626007 is enabled.
xtrabackup: cd to /data/backups/base
xtrabackup: This target seems to be already prepared with --apply-log-only.
xtrabackup: xtrabackup_logfile detected: size=2097152, start_lsn=(4124244)
...
xtrabackup: page size for /tmp/backups/inc1/ibdata1.delta is 16384 bytes
Applying /tmp/backups/inc1/ibdata1.delta to ./ibdata1...
...
161011 12:45:56 completed OK!
Again, the LSN should match what you saw from your earlier inspection of the first incremental backup. If
you restore the files from /data/backups/base , you should see the state of the database as of the first
incremental backup.
Warning
Percona XtraBackup does not support using the same incremental backup directory to prepare two copies of
backup. Do not run --prepare with the same incremental backup directory (the value of –incremental-dir) more
than once.
Preparing the second incremental backup is a similar process: apply the deltas to the (modified) base
backup, and you will roll its data forward in time to the point of the second incremental backup:
Note
--apply-log-only should be used when merging the incremental backups except the last one. That’s why the
previous line does not contain the --apply-log-only option. Even if the --apply-log-only was used on the last
step, backup would still be consistent but in that case server would perform the rollback phase.
Once prepared incremental backups are the same as the full backups, and they can be restored in the
same way.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
Note
Starting with Percona XtraBackup 8.0.31-24 using qpress/QuickLZ to compress backups is deprecated and may be
removed in future versions. We recommend using either LZ4 or Zstandard ( ZSTD ) compression algorithms.
To make a compressed backup, use the --compress option along with the --backup and --target-dir
options.
By default, the --compress option uses the qpress tool that you can install with the percona-release
package configuration tool as follows:
Note
If Percona XtraBackup is intended to be used in combination with the upstream MySQL Server, you only need
to enable the tools repository: percona-release enable-only tools .
quicklz
To compress files using the quicklz compression algorithm, use --compress option:
lz4
To compress files using the lz4 compression algorithm, set --compress option to lz4 :
Zstandard (ZSTD)
The Zstandard (ZSTD) compression algorithm is a tech preview feature. Before using ZSTD in production, we
recommend that you test restoring production from physical backups in your environment, and also use the
alternative backup method for redundancy.
Percona XtraBackup 8.0.30-23 adds support for the Zstandard (ZSTD) compression algorithm. ZSTD is a fast
lossless compression algorithm that targets real-time compression scenarios and better compression
ratios.
To compress files using the ZSTD compression algorithm, set --compress option to zstd :
You can specify ZSTD compression level with the --compress-zstd-level(=#) option. The default value is 1 .
If you want to speed up the compression you can use the parallel compression, which can be enabled with
--compress-threads option. Following example will use four threads for compression:
Expected output
{.text .no-copy}
...
170223 13:00:38 [01] Compressing ./test/sbtest1.frm to /tmp/compressed/test/sbtest1.frm.qp
170223 13:00:38 [01] ...done
170223 13:00:38 [01] Compressing ./test/sbtest2.frm to /tmp/compressed/test/sbtest2.frm.qp
170223 13:00:38 [01] ...done
...
170223 13:00:39 [00] Compressing xtrabackup_info
170223 13:00:39 [00] ...done
xtrabackup: Transaction log of lsn (9291934) to (9291934) was copied.
170223 13:00:39 completed OK!
Before you can prepare the backup you’ll need to uncompress all the files. Percona XtraBackup has
implemented --decompress option that can be used to decompress the backup.
Note
--parallel can be used with --decompress option to decompress multiple files simultaneously.
Percona XtraBackup does not automatically remove the compressed files. In order to clean up the backup
directory you should use --remove-original option. Even if they’re not removed these files will not be copied/
moved over to the datadir if --copy-back or --move-back are used.
When the files are uncompressed you can prepare the backup with the --prepare option:
Confirmation message
{.text .no-copy}
InnoDB: Starting shutdown...
InnoDB: Shutdown completed; log sequence number 9293846
170223 13:39:31 completed OK!
xtrabackup has a --copy-back option, which performs the restoration of a backup to the server’s datadir:
It will copy all the data-related files back to the server’s datadir, determined by the server’s my.cnf
configuration file. You should check the last line of the output for a success message:
Expected output
{.text .no-copy}
170223 13:49:13 completed OK!
You should check the file permissions after copying the data back. You may need to adjust them with
something like:
Now that the datadir contains the restored data. You are ready to start the server.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
Warning
Restoring partial backups should be done by importing the tables, not by using the –copy-back option. It is not
recommended to run incremental backups after running a partial backup.
Although there are some scenarios where restoring can be done by copying back the files, this may lead to
database inconsistencies in many cases and it is not a recommended way to do it.
For the purposes of this manual page, we will assume that there is a database named test which contains
tables named t1 and t2 .
Warning
If any of the matched or listed tables is deleted during the backup, xtrabackup will fail.
There are multiple ways of specifying which part of the whole data is backed up:
The first method involves the xtrabackup –tables option. The option’s value is a regular expression that is
matched against the fully-qualified database name and table name using the databasename.tablename
format.
To back up only tables in the test database, use the following command:
The --tables-file option specifies a file that can contain multiple table names, one table name per line in
the file. Only the tables named in the file will be backed up. Names are matched exactly, case-sensitive, with
no pattern or regular expression matching. The table names must be fully-qualified in
databasename.tablename format.
The ` –databases` option accepts a space-separated list of the databases and tables to backup in the
databasename[.tablename] format. In addition to this list, make sure to specify the mysql , sys , and
performance_schema databases. These databases are required when restoring the databases using
xtrabackup –copy-back.
Note
Tables processed during the –prepare step may also be added to the backup even if they are not explicitly listed
by the parameter if they were created after the backup started.
The –databases-file option specifies a file that can contain multiple databases and tables in the
databasename[.tablename] format, one element name per line in the file. Names are matched exactly, case-
sensitive, with no pattern or regular expression matching.
Note
Tables processed during the –prepare step may also be added to the backup even if they are not explicitly listed
by the parameter if they were created after the backup started.
The procedure is analogous to restoring individual tables: apply the logs and use the --export option:
When you use the --prepare option on a partial backup, you will see warnings about tables that don’t exist.
This is because these tables exist in the data dictionary inside InnoDB, but the corresponding .ibd files don’t
exist. They were not copied into the backup directory. These tables will be removed from the data dictionary,
and when you restore the backup and start InnoDB, they will no longer exist and will not cause any errors or
warnings to be printed to the log file.
Could not find any file associated with the tablespace ID: 5
Use --innodb-directories to find the tablespace files. If that fails then use -–innodb-force-recovery=1 to
ignore this and to permanently lose all changes to the missing tablespace(s).
Restoring should be done by restoring individual tables in the partial backup to the server.
It can also be done by copying back the prepared backup to a “clean” datadir (in that case, make sure to
include the mysql database) to the datadir you are moving the backup to. A system database can be
created with the following:
Once you start the server, you may see mysql complaining about missing tablespaces:
Expected output
{.text .no-copy}
2021-07-19T12:42:11.077200Z 1 [Warning] [MY-012351] [InnoDB] Tablespace 4, name 'test1/t1', file './d2/
test1.ibd' is missing!
2021-07-19T12:42:11.077300Z 1 [Warning] [MY-012351] [InnoDB] Tablespace 4, name 'test1/t1', file './d2/
test1.ibd' is missing!
In order to clean the orphan database from the data dictionary, you must manually create the missing
database directory and then DROP this database from the server.
$ mkdir /var/lib/mysql/test1/d2
Expected output
{.text .no-copy}
Query OK, 2 rows affected (0.5 sec)
Contact us
For paid support and managed or consulting services , contact Percona Sales.
7. User’s manual
xtrabackup
xbcrypt
xbstream
xbcloud
The recommended way to take the backup is by using the xtrabackup script. For more information on script
options, see xtrabackup.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
8. Advanced features
The image below shows how throttling works when --throttle is set to 1 .
When specified with the --backup option, this option limits the number of pairs of read-and-write operations
per second that xtrabackup will perform. If you are creating an incremental backup, then the limit is the
number of read I/O operations per second.
By default, there is no throttling, and xtrabackup reads and writes data as quickly as it can. If you set too
strict of a limit on the IOPS, the backup might be so slow that it will never catch up with the transaction logs
that InnoDB is writing, so the backup might never complete.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
For an authenticated user or application to access an encrypted tablespace, InnoDB uses the master
encryption key to decrypt the tablespace key. The master encryption key is stored in a keyring. xtrabackup
supports two keyring plugins: keyring_file , and keyring_vault . These plugins are installed into the plugin
directory.
Percona XtraBackup 8.0.25-17 adds support for the keyring_file component, which is part of the
component-based infrastructure MySQL which extends the server capabilities. The component is stored in
the plugin directory.
See a comparison of keyring components and keyring plugins for more information.
Percona XtraBackup 8.0.27-19 adds support for the Key Management Interoperability Protocol (KMIP) which
enables the communication between the key management system and encrypted database server.
Percona XtraBackup 8.0.28-21 adds support for the Amazon Key Management Service (AWS KMS). AWS KMS
is cloud-based encryption and key management service. The keys and functionality can be used for other
AWS services or your applications that use AWS. No configuration is required to back up a server with AWS
KMS-enabled encryption.
After xtrabackup takes the backup, the following message confirms the action:
Confirmation message
{.text .no-copy}
xtrabackup: Transaction log of lsn (5696709) to (5696718) was copied.
160401 10:25:51 completed OK!
Warning
xtrabackup does not copy the keyring file into the backup directory. To prepare the backup, you must copy the
keyring file manually.
After xtrabackup takes the backup, the following message confirms the action:
Confirmation message
{.text .no-copy}
InnoDB: Shutdown completed; log sequence number 5697064
160401 10:34:28 completed OK!
The backup is now prepared and can be restored with the --copy-back option. In case the keyring has been
rotated, you must restore the keyring used when the backup was taken and prepared.
After xtrabackup completes the action, the following message confirms the action:
Confirmation message
{.text .no-copy}
xtrabackup: Transaction log of lsn (5696709) to (5696718) was copied.
160401 10:25:51 completed OK!
To prepare the backup, xtrabackup must access the keyring. xtrabackup does not communicate with the
MySQL server or read the default my.cnf configuration file. Specify the keyring settings in the command line:
Review using the keyring vault plugin for a description of keyring vault plugin settings.
After xtrabackup completes the action, the following message confirms the action:
Confirmation message
{.text .no-copy}
InnoDB: Shutdown completed; log sequence number 5697064
160401 10:34:28 completed OK!
The backup is now prepared and can be restored with the --copy-back option:
A component is not loaded with the --early_plugin_load option. The server uses a manifest to load the
component and the component has its own configuration file. See the MySQL documentation on the
component installation for more information.
An example of ./bin/mysqld.my :
{
"components": "file://component_keyring_file"
}
An example of /lib/plugin/component_keyring_file.cnf :
{
"path": "/var/lib/mysql-keyring/keyring_file", "read_only": false
}
For more information, see Keyring Component Installation and Using the keyring_file File-Based Keyring
Plugin.
The component has no special requirements for backing up a database that contains encrypted InnoDB
tablespaces.
After xtrabackup completes the action, the following message confirms the action:
Confirmation message
{.text .no-copy}
xtrabackup: Transaction log of lsn (5696709) to (5696718) was copied.
160401 10:25:51 completed OK!
Warning
xtrabackup does not copy the keyring file into the backup directory. To prepare the backup, you must copy the
keyring file manually.
xtrabackup attempts to read xtrabackup_component_keyring_file.cnf . You can assign another keyring file
component configuration by passing the --component-keyring-file-config option.
After xtrabackup completes preparing the backup, the following message confirms the action:
Confirmation message
{.text .no-copy}
InnoDB: Shutdown completed; log sequence number 5697064
160401 10:34:28 completed OK!
The backup is prepared. To restore the backup use the --copy-back option. If the keyring has been rotated,
you must restore the specific keyring used to take and prepare the backup.
The process of taking incremental backups with InnoDB tablespace encryption is similar to taking the
Incremental Backups with unencrypted tablespace. The keyring-file component should not used in
production or for regulatory compliance.
To make an incremental backup, begin with a full backup. The xtrabackup binary writes a file called
xtrabackup_checkpoints into the backup’s target directory. This file contains a line showing the to_lsn , which
is the database’s LSN at the end of the backup. First you need to create a full backup with the following
command:
In order to prepare the backup, you must make a copy of the keyring file yourself. xtrabackup does not copy
the keyring file into the backup directory. Restoring the backup after the keyring has been changed causes
errors like ERROR 3185 (HY000): Can't find master key from keyring, please check keyring plugin is loaded.
when the restore process tries accessing an encrypted table.
If you look at the xtrabackup_checkpoints file, you should see the output similar to the following:
Expected output
{.text .no-copy}
backup_type = full-backuped
from_lsn = 0
to_lsn = 7666625
last_lsn = 7666634
compact = 0
recover_binlog_info = 1
Now that you have a full backup, you can make an incremental backup based on it. Use a command such
as the following:
To prepare the backup, you must copy the keyring file manually. xtrabackup does not copy the keyring file
into the backup directory.
If the keyring has not been rotated you can use the same as the one you’ve backed-up with the base
backup. If the keyring has been rotated, or you have upgraded the plugin to a component, you must back
up the keyring file. Otherwise, you cannot prepare the backup.
The /data/backups/inc1/ directory should now contain delta files, such as ibdata1.delta and test/
table1.ibd.delta . These represent the changes since the LSN 7666625 . If you examine the
xtrabackup_checkpoints file in this directory, you should see the output similar to the following:
Expected output
{.text .no-copy}
backup_type = incremental
from_lsn = 7666625
to_lsn = 8873920
last_lsn = 8873929
compact = 0
recover_binlog_info = 1
Use this directory as the base for yet another incremental backup:
The --prepare step for incremental backups is not the same as for normal backups. In normal backups, two
types of operations are performed to make the database consistent: committed transactions are replayed
from the log file against the data files, and uncommitted transactions are rolled back. You must skip the
rollback of uncommitted transactions when preparing a backup, because transactions that were
uncommitted at the time of your backup may be in progress, and it’s likely that they will be committed in
the next incremental backup. You should use the --apply-log-only option to prevent the rollback phase.
If you do not use the --apply-log-only option to prevent the rollback phase, then your incremental backups
are useless. After transactions have been rolled back, further incremental backups cannot be applied.
Beginning with the full backup you created, you can prepare it and then apply the incremental differences
to it. Recall that you have the following backups:
/data/backups/base
/data/backups/inc1
/data/backups/inc2
To prepare the base backup, you need to run --prepare as usual, but prevent the rollback phase:
Expected output
{.text .no-copy}
InnoDB: Shutdown completed; log sequence number 7666643
InnoDB: Number of pools: 1
160401 12:31:11 completed OK!
To apply the first incremental backup to the full backup, you should use the following command:
The backup should be prepared with the keyring file and type that was used when backup was being taken.
This means that if the keyring has been rotated, or you have upgraded from a plugin to a component
between the base and incremental backup that you must use the keyring that was in use when the first
incremental backup has been taken.
Preparing the second incremental backup is a similar process: apply the deltas to the (modified) base
backup, and you will roll its data forward in time to the point of the second incremental backup:
Use --apply-log-only when merging all incremental backups except the last one. That’s why the previous
line does not contain the --apply-log-only option. Even if the --apply-log-only was used on the last step,
backup would still be consistent but in that case server would perform the rollback phase.
The backup is now prepared and can be restored with --copy-back option. In case the keyring has been
rotated you’ll need to restore the keyring which was used to take and prepare the backup.
Percona XtraBackup 8.0.27-19 adds support for the Key Management Interoperability Protocol (KMIP) which
enables the communication between the key management system and encrypted database server.
Percona XtraBackup has no special requirements for backing up a database that contains encrypted
InnoDB tablespaces.
When preparing the backup, Percona XtraBackup connects to the KMIP server with the settings from the
xtrabackup_component_keyring_kmip.cnf file.
While the described restore method works, this method requires access to the same keyring that the server
is using. It may not be possible if the backup is prepared on a different server or at a much later time, when
keys in the keyring are purged, or, in the case of a malfunction, when the keyring vault server is not available
at all.
The --transition-key=<passphrase> option should be used to make it possible for xtrabackup to process the
backup without access to the keyring vault server. In this case, xtrabackup derives the AES encryption key
from the specified passphrase and will use it to encrypt tablespace keys of tablespaces that are being
backed up.
The following example illustrates how the backup can be created in this case:
xtrabackup scrapes --transition-key so that its value is not visible in the ps command output.
When restoring a backup you will need to generate a new master key. Here is the example for keyring_file
plugin or component:
xtrabackup will generate a new master key, store it in the target keyring vault server and re-encrypt the
tablespace keys using this key.
Finally, there is an option to store a transition key in the keyring. In this case, xtrabackup will need to access
the same keyring file or vault server during prepare and copy-back but does not depend on whether the
server keys have been purged.
In this scenario, the three stages of the backup process look as follows.
• Backup
{.bash data-prompt="$"}
$ xtrabackup --backup --user=root -p --target-dir=/data/backup \
--generate-transition-key
• Prepare
• keyring_file variant:
{.bash data-prompt="$"}
$ xtrabackup --prepare --target-dir=/data/backup \
--keyring-file-data=/var/lib/mysql-keyring/keyring
• keyring_vault variant:
{.bash data-prompt="$"}
$ xtrabackup --prepare --target-dir=/data/backup \
--keyring-vault-config=/etc/vault.cnf
• Copy-back
• keyring_file variant:
{.bash data-prompt="$"}
$ xtrabackup --copy-back --target-dir=/data/backup --datadir=/data/mysql \
--generate-new-master-key --keyring-file-data=/var/lib/mysql-keyring/keyring
• keyring_vault variant:
{.bash data-prompt="$"}
$ xtrabackup --copy-back --target-dir=/data/backup --datadir=/data/mysql \
--generate-new-master-key --keyring-vault-config=/etc/vault.cnf
Contact us
For paid support and managed or consulting services , contact Percona Sales.
To make an encrypted backup the following options need to be specified (options --encrypt-key and --
encrypt-key-file are mutually exclusive, i.e. just one of them needs to be provided):
• --encrypt
• --encrypt-key
• --encrypt-key-file
Both the --encrypt-key option and --encrypt-key-file option can be used to specify the encryption key. An
encryption key can be generated with a command like openssl rand -base64 32
U2FsdGVkX19VPN7VM+lwNI0fePhjgnhgqmDBqbF3Bvs=
Example of the xtrabackup command using the --encrypt-key should look like this:
Note
Depending on the text editor that you use to make the KEYFILE , the editor can automatically insert the CRLF (end
of line) character. This will cause the key size to grow and thus making it invalid. The suggested way to create the
file is by using the command line: echo -n “U2FsdGVkX19VPN7VM+lwNI0fePhjgnhgqmDBqbF3Bvs=” > /data/backups/
keyfile .
Two new options are available for encrypted backups that can be used to speed up the encryption process.
These are --encrypt-threads and --encrypt-chunk-size . By using the --encrypt-threads option multiple
threads can be specified to be used for encryption in parallel. Option --encrypt-chunk-size can be used to
specify the size (in bytes) of the working encryption buffer for each encryption thread (default is 64K).
Backups can be decrypted with The xbcrypt binary. The following one-liner can be used to encrypt the
whole folder:
Percona XtraBackup --decrypt option has been implemented that can be used to decrypt the backups:
Percona XtraBackup doesn’t automatically remove the encrypted files. In order to clean up the backup
directory users should remove the \*.xbcrypt files.
Note
--parallel can be used with --decrypt option to decrypt multiple files simultaneously.
After the backups have been decrypted, they can be prepared in the same way as the standard full
backups with the --prepare option:
xtrabackup offers the --copy-back option to restore a backup to the server’s datadir:
It will copy all the data-related files back to the server’s datadir, determined by the server’s my.cnf
configuration file. You should check the last line of the output for a success message:
Expected output
{.text .no-copy}
150318 11:08:13 xtrabackup: completed OK!
Contact us
For paid support and managed or consulting services , contact Percona Sales.
If the buffer restore option is enabled in my.cnf buffer pool will be in the warm state after backup is restored.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
Note that the binary log contains the operations that modified the database from a point in the past. You
need a full datadir as a base, and then you can apply a series of operations from the binary log to make the
data match what it was at the point in time you want.
For more details on these procedures, see Creating a backup and Preparing a backup.
Now, suppose that some time has passed, and you want to restore the database to a certain point in the
past, having in mind that there is the constraint of the point where the snapshot was taken.
To find out what is the situation of binary logging in the server, execute the following queries:
Expected output
{.text .no-copy}
+------------------+-----------+
| Log_name | File_size |
+------------------+-----------+
| mysql-bin.000001 | 126 |
| mysql-bin.000002 | 1306 |
| mysql-bin.000003 | 126 |
| mysql-bin.000004 | 497 |
+------------------+-----------+
and
Expected output
{.text .no-copy}
+------------------+----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000004 | 497 | | |
+------------------+----------+--------------+------------------+
The first query will tell you which files contain the binary log and the second one which file is currently being
used to record changes, and the current position within it. Those files are stored usually in the datadir
(unless other location is specified when the server is started with the --log-bin= option).
To find out the position of the snapshot taken, see the xtrabackup_binlog_info at the backup’s directory:
$ cat /path/to/backup/xtrabackup_binlog_info
Expected output
{.text .no-copy}
mysql-bin.000003 57
This will tell you which file was used at moment of the backup for the binary log and its position. That
position will be the effective one when you restore the backup:
As the restoration will not affect the binary log files (you may need to adjust file permissions, see Restoring a
Backup), the next step is extracting the queries from the binary log with mysqlbinlog starting from the
position of the snapshot and redirecting it to a file
Note that if you have multiple files for the binary log, as in the example, you have to extract the queries with
one process, as shown above.
Inspect the file with the queries to determine which position or date corresponds to the point-in-time
wanted. Once determined, pipe it to the server. Assuming the point is 11-12-25 01:00:00 :
Contact us
For paid support and managed or consulting services , contact Percona Sales.
To generate an .ibd file in the target directory, create the table using the innodb_file_per_table mode:
During the --prepare step, add the --export option to the command. For example:
When restoring an encrypted InnoDB tablespace table, add the keyring file:
The following files are the only files required to import the table into a server running Percona Server for
MySQL with XtraDB or MySQL 8.0. If the server uses InnoDB Tablespace Encryption, add the .cfp file, which
contains the transfer key and an encrypted tablespace key.
/data/backups/mysql/test/export_test.ibd
/data/backups/mysql/test/export_test.cfg
On the destination server running Percona Server for MySQL with XtraDB or MySQL 8.0, create a table with the
same structure, and then perform the following steps:
1. Run the ALTER TABLE test.export_test DISCARD TABLESPACE; command. If you see the following error
message:
Error message
{.text .no-copy}
ERROR 1809 (HY000): Table 'test/export_test' in system tablespace
enable innodb_file_per_table option on the server and create the table again.
{.bash data-prompt="$"}
$ set global innodb_file_per_table=ON;
2. Copy the exported files to the test/ subdirectory of the destination server’s data directory.
The table is imported, and you can run a SELECT to see the imported data.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
Percona XtraBackup 8.0.30-23 adds support for the Smart memory estimation feature. With this feature,
Percona XtraBackup computes the memory required for prepare phase, while copying redo log entries
during the backup phase. Percona XtraBackup also considers the number of InnoDB pages to be fetched
from the disk.
• Creates a backup
To create a backup, Percona XtraBackup copies your InnoDB data files. While copying the files, Percona
XtraBackup runs a background process that watches the InnoDB redo log, also called the transaction
log, and copies changes from it.
• Prepares a backup
During the prepare phase, Percona XtraBackup performs crash recovery against the copied data files
using the copied transaction log file. Percona XtraBackup reads all the redo log entries into memory,
categorizes them by space id and page id, reads the relevant pages into memory, and checks the log
sequence number (LSN) on the page and on the redo log record. If the redo log LSN is more recent than
the page LSN, Percona XtraBackup applies the redo log changes to the page.
To prepare a backup, Percona Xtrabackup uses InnoDB Buffer Pool memory. Percona Xtrabackup
reserves memory to load 256 pages into the buffer pool. The remaining memory is used for hashing/
categorizing the redo log entries.
The available memory is controlled by the --use-memory option. If the available memory on the buffer
pool is insufficient, the work is performed in multiple batches. After the batch is processed, the memory
is freed to release space for the next batch. This process greatly impacts performance as an InnoDB
page holds data from multiple rows. If a change on a page happens in different batches, that page is
fetched and evicted numerous times.
In the prepare phase, Percona XtraBackup checks the server's available free memory and uses that
memory up to the limit specified in the --use-free-memory-pct option to run --prepare . Due to backward
compatibility, the default value for the --use-free-memory-pct option is 0 (zero), which defines the option as
disabled. For example, if you set --use-free-memory-pct=50 , then 50% of the free memory is used to prepare
a backup.
Starting with Percona XtraBackup 8.0.32-26, you can enable or disable the memory estimation during the
backup phase with the --estimate-memory option. The default value is OFF . Enable the memory estimation
with --estimate-memory=ON :
In the prepare phase, enable the --use-free-memory-pct option by specifying the percentage of free
memory to be used to prepare a backup. The --use-free-memory-pct value must be larger than 0.
For example:
The examples of how Smart memory estimation can improve the time spent on prepare in different versions
of Percona XtraBackup:
We back up 16, 32, and 64 tables using sysbench. Each set contains 1M rows. In the backup phase, we enable
Smart memory estimation with --estimate-memory=ON . In the prepare phase, we set
--use-free-memory-pct=50 , and Percona XtraBackup uses 50% of the free memory to prepare a backup. The
backup is run on an ec2 c4.8xlarge instance (36 vCPUs / 60GB memory / General Purpose SSD (gp2)).
We back up 16, 32, and 64 tables using sysbench. Each set contains 1M rows. In the prepare phase, we set --
use-free-memory-pct=50 , and Percona XtraBackup uses 50% of the free memory to prepare a backup. The
backup is run on an ec2 c4.8xlarge instance (36 vCPUs / 60GB memory / General Purpose SSD (gp2)).
The following table shows the backup details (all measurements are in Gigabytes):
• Size of XtraBackup log - the size of Percona XtraBackup log file (redo log entries copied during the
backup)
• Size of backup - the size of the resulting backup folder
Prepare executed without Smart memory estimation uses the default of 128MB for the buffer pool.
Note
The following results are based on tests in a specific environment. Your results may vary.
• 16 tables result - prepare time dropped to ~5.7% of the original time. An improvement in recovery time
of about 17x.
• 32 tables result - prepare time dropped to ~8,2% of the original time. An improvement in recovery time
of about 12x.
• 64 tables result - prepare time dropped to ~9.9% of the original time. An improvement in recovery time
of about 10x.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
You can find the binary log position corresponding to a backup after the backup has been taken. If your
backup is from a server with binary logging enabled, xtrabackup creates a file named
xtrabackup_binlog_info in the target directory. This file contains the binary log file name and position of the
exact point when the backup was taken.
{.text .no-copy}
210715 14:14:59 Backup created in directory '/backup/'
MySQL binlog position: filename 'binlog.000002', position '156'
. . .
210715 14:15:00 completed OK!
Note
As of Percona XtraBackup 8.0.26-18.0, xtrabackup no longer creates the xtrabackup_binlog_pos_innodb file. This
change is because MySQL and Percona Server no longer update the binary log information on global transaction
system section of ibdata . You should rely on xtrabackup_binlog_info regardless of the storage engine in use.
To perform a point-in-time recovery from an xtrabackup backup, you should prepare and restore the
backup, and then replay binary logs from the point shown in the xtrabackup_binlog_info file.
To set up a new replica, you should prepare the backup, and restore it to the data directory of your new
replication replica. If you are using version 8.0.22 or earlier, in your CHANGE MASTER TO command, use the
binary log filename and position shown in the xtrabackup_binlog_info file to start replication.
If you are using 8.0.23 or later, use the CHANGE_REPLICATION_SOURCE_TO and the appropriate options.
CHANGE_MASTER_TO is deprecated.
A more detailed procedure is found in How to setup a replica for replication in 6 simple steps with Percona
XtraBackup.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
In earlier versions, the error logs did not have a standard structure. Notice in the following examples the
variance in the log statements:
• The backup log statement header has the name of the module, xtrabackup , which generated the
statement but no timestamp:
Expected output
{.text .no-copy}
./bin/xtrabackup version 8.0.27-19 based on MySQL server 8.0.27 Linux (x86_64) (revision id: b0f75188ca3)
• The copy-back log statement has a timestamp but no module name. The timestamp is a mix of UTC
and the local timezone.
• The following prepare log statements do not have header information, which makes diagnosing an
issue more difficult.
Starting in Percona XtraBackup 8.0.28-20, changes have been made to improve the log statements. The
improved log structure is displayed in the backup, prepare, move-back/copy-back error logs.
An example of a prepare log that is generated with the improved structure. The uniformity of the headers
makes it easier to follow an operation’s progress or review the log to diagnose issues.
Expected output
{.text .no-copy}
2022-03-22T19:15:36.142247+05:30 0 [Note] [MY-011825] [Xtrabackup] This target seems to be not prepared yet.
2022-03-22T19:15:36.142792+05:30 0 [Note] [MY-013251] [InnoDB] Number of pools: 1
2022-03-22T19:15:36.149212+05:30 0 [Note] [MY-011825] [Xtrabackup] xtrabackup_logfile detected: size=8388608,
start_lsn=(33311656)
2022-03-22T19:15:36.149998+05:30 0 [Note] [MY-011825] [Xtrabackup] using the following InnoDB configuration
for recovery:
2022-03-22T19:15:36.150023+05:30 0 [Note] [MY-011825] [Xtrabackup] innodb_data_home_dir = .
2022-03-22T19:15:36.150036+05:30 0 [Note] [MY-011825] [Xtrabackup] innodb_data_file_path =
ibdata1:12M:autoextend
2022-03-22T19:15:36.150078+05:30 0 [Note] [MY-011825] [Xtrabackup] innodb_log_group_home_dir = .
2022-03-22T19:15:36.150095+05:30 0 [Note] [MY-011825] [Xtrabackup] innodb_log_files_in_group = 1
2022-03-22T19:15:36.150111+05:30 0 [Note] [MY-011825] [Xtrabackup] innodb_log_file_size = 8388608
2022-03-22T19:15:36.151667+05:30 0 [Note] [MY-011825] [Xtrabackup] inititialize_service_handles suceeded
2022-03-22T19:15:36.151903+05:30 0 [Note] [MY-011825] [Xtrabackup] using the following InnoDB configuration
for recovery:
2022-03-22T19:15:36.151926+05:30 0 [Note] [MY-011825] [Xtrabackup] innodb_data_home_dir = .
2022-03-22T19:15:36.151954+05:30 0 [Note] [MY-011825] [Xtrabackup] innodb_data_file_path =
ibdata1:12M:autoextend
2022-03-22T19:15:36.151976+05:30 0 [Note] [MY-011825] [Xtrabackup] innodb_log_group_home_dir = .
2022-03-22T19:15:36.151991+05:30 0 [Note] [MY-011825] [Xtrabackup] innodb_log_files_in_group = 1
2022-03-22T19:15:36.152004+05:30 0 [Note] [MY-011825] [Xtrabackup] innodb_log_file_size = 8388608
2022-03-22T19:15:36.152021+05:30 0 [Note] [MY-011825] [Xtrabackup] Starting InnoDB instance for recovery.
2022-03-22T19:15:36.152035+05:30 0 [Note] [MY-011825] [Xtrabackup] Using 104857600 bytes for buffer pool (set
by --use-memory parameter)
Contact us
For paid support and managed or consulting services , contact Percona Sales.
9. Security
You find the current state of the Percona XtraBackup file with the following command:
Expected output
{.text .no-copy}
-rwxr-xr-x. root root system_u:object_r:bin_t:s0 xtrabackup
• user (root)
• role (object_r)
• type (bin_t)
• level (s0)
The unconfined domain supports the network-facing services, which are protected by SELinux. These
domains are not exposed. In this configuration, SELinux protects against remote intrusions but local
intrusions, which require local access, are not confined.
Percona XtraBackup works locally. The service is not network-facing and cannot be exploited externally. The
service interacts only with the local user, who provides the parameters. Percona XtraBackup requires access
to the target-dir location.
You can modify your security configuration to confine Percona XtraBackup. The first question is where to
store the backup files. The service requires read and write access to the selected location.
• Allow Percona XtraBackup to write to any location. The user provides any path to the target-dir
parameter.
• Allow Percona XtraBackup to write to a specific location, such as /backups or the user’s home directory.
The first option opens the entire system to read and write. Select the second option to harden your security.
To work with policies, you must install the SELinux tools. To find which package provides the semanage
command and install the package. The following is an example on CentOS 7.
Expected output
{.text .no-copy}
...
policycoreutils-python-2.5-34.el7.x86_64 : SELinux policy core python utilities
...
Expected output
{.text .no-copy}
...
policycoreutils-python-utils-2.8-16.1.el8.noarch : SELinux policy core python utilities
Use a modular approach to create an SELinux policy. Create a policy module to manage XtraBackup. You
must create a .te file for type enforcement, and an optional .fc file for the file contexts.
Use $ ps -efZ | grep xtrabackup to verify the service is not confined by SELinux.
Create the xtrabackup.fc file and add content. This file defines the security contexts.
/usr/bin/xtrabackup -- gen_context(system_u:object_r:xtrabackup_exec_t,s0)
/usr/bin/xbcrypt -- gen_context(system_u:object_r:xtrabackup_exec_t,s0)
/usr/bin/xbstream -- gen_context(system_u:object_r:xtrabackup_exec_t,s0)
/usr/bin/xbcloud -- gen_context(system_u:object_r:xtrabackup_exec_t,s0)
/backups(/.*)? system_u:object_r:xtrabackup_data_t:s0
Note
If you are using the /backups directory you must have the last line. If you are storing the backups in the user’s
home directory, you can omit this line.
https://github.com/percona/percona-xtrabackup/tree/8.0/packaging/percona/selinx
Note
In the file, the sections in bold should be modified for your system. The fc file can also be downloaded from the
same location.
$ semodule -i xtrabackup.pp
Tag the PXB binaries with the proper SELinux tags, such as xtrabackup_exec_t .
$ restorecon -v /usr/bin/*
If you store your backups at /backups , restore the tag in that location:
$ restorecon -v /backups
Note
Remember to add the standard Linux DAC permissions for this directory.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
Percona XtraBackup does not have a profile and is not confined by AppArmor.
For a list of common AppArmor commands, see Percona Server for MySQL - AppArmor
https://github.com/percona/percona-xtrabackup/tree/8.0/packaging/percona/apparmor/apparmor.d
The following profile sections should be updated with your system information, such as location of the
backup destination directory.
Expected output
```{.text .no-copy}
Contact us
For paid support and managed or consulting services , contact Percona Sales.
• there is a line starting with tcp (the server is indeed accepting TCP connections)
• the first address ( 0.0.0.0:3306 in this example) is different from 127.0.0.1:3306 (the bind address is
not localhost’s).
In the first case, the first place to look is the my.cnf file. If you find the option skip-networking , comment it
out or just delete it. Also check that if the variable bind_address is set, then it should not be set to localhost’s
but to the host’s IP. Then restart the MySQL server and check it again with netstat . If the changes had no
effect, then you should look at your distribution’s startup scripts (like rc.mysqld ). You should comment out
flags like --skip-networking and/or change the bind-address .
After you get the server listening to remote TCP connections properly, the last thing to do is checking that
the port (3306 by default) is indeed open. Check your firewall configurations ( iptables -L ) and that you are
allowing remote hosts on that port (in /etc/hosts.allow ).
And we’re done! We have a MySQL server running which is able to communicate with the world through TCP/
IP.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
Note
As of Percona XtraBackup 8.0.30 the --stats option is deprecated and may be removed in future releases. We no
longer support using the --stats option in Percona XtraBackup 8.0.29 and older versions.
The xtrabackup binary can analyze InnoDB data files in read-only mode to give statistics about them. To do
this, you should use the --stats option. You can combine this with the --tables option to limit the files to
examine. It also uses the --use-memory option.
You can perform the analysis on a running server, with some chance of errors due to the data being
changed during analysis. Or, you can analyze a backup copy of the database. Either way, to use the
statistics feature, you need a clean copy of the database including correctly sized log files, so you need to
execute with --prepare twice to use this functionality on a backup.
Expected output
{.text .no-copy}
<INDEX STATISTICS>
table: test/table1, index: PRIMARY, space id: 12, root page 3
estimated statistics in dictionary:
key vals: 25265338, leaf pages 497839, size pages 498304
real statistics:
level 2 pages: pages=1, data=5395 bytes, data/pages=32%
level 1 pages: pages=415, data=6471907 bytes, data/pages=95%
leaf pages: recs=25958413, pages=497839, data=7492026403 bytes, data/pages=91%
• The first line simply shows the table and index name and its internal identifiers. If you see an index
named GEN_CLUST_INDEX , that is the table’s clustered index, automatically created because you did not
explicitly create a PRIMARY KEY .
• The estimated statistics in dictionary information is similar to the data that’s gathered through
ANALYZE TABLE inside of InnoDB to be stored as estimated cardinality statistics and passed to the query
optimizer.
• The real statistics information is the result of scanning the data pages and computing exact
information about the index.
• The level <X> pages : output means that the line shows information about pages at that level in the
index tree. The larger <X> is, the farther it is from the leaf pages, which are level 0. The first line is the
root page.
• The leaf pages output shows the leaf pages, of course. This is where the table’s data is stored.
• The external pages : output (not shown) shows large external pages that hold values too long to fit in
the row itself, such as long BLOB and TEXT values.
The following script can be used to summarize and tabulate the output of the statistics information:
Expected output
my $PG_SIZE = 16_384; # InnoDB defaults to 16k pages, change if needed. my ($cur_idx, $cur_tbl); my
(%idx_stats, %tbl_stats); my ($max_tbl_len, $max_idx_len) = (0, 0); while ( my $line = <> ) { if ( my ($t, $i) = $line
=~ m/table: (.), index: (.), space id:/ ) { $t =~ s!/!.!; $cur_tbl = $t; $cur_idx = $i; if ( length($i) > $max_idx_len ) {
$max_idx_len = length($i); } if ( length($t) > $max_tbl_len ) { $max_tbl_len = length($t); } } elsif ( my ($kv, $lp,
$sp) = $line =~ m/key vals: (\d+), \D(\d+), \D(\d+)/ ) { @{$idx_stats{$cur_tbl}->{$cur_idx}}{qw(est_kv est_lp
est_sp)} = ($kv, $lp, $sp); $tbl_stats{$cur_tbl}->{est_kv} += $kv; $tbl_stats{$cur_tbl}->{est_lp} += $lp;
$tbl_stats{$cur_tbl}->{est_sp} += $sp; } elsif ( my ($l, $pages, $bytes) = $line =~ m/(?:level (\d+)|leaf)
pages:.*pages=(\d+), data=(\d+) bytes/ ) { $l ||= 0; $idx_stats{$cur_tbl}->{$cur_idx}->{real_pages} +=
$pages; $idx_stats{$cur_tbl}->{$cur_idx}->{real_bytes} += $bytes; $tbl_stats{$cur_tbl}->{real_pages} +=
$pages; $tbl_stats{$cur_tbl}->{real_bytes} += $bytes; } }
my $row_fmt = "%${max_tbl_len}s %${max_idx_len}s %9d %10d %9.1f%%\n"; foreach my $t ( sort keys %tbl_stats
) { my $tbl = $tbl_stats{$t}; printf $row_fmt, $t, "", $tbl->{est_sp}, $tbl->{est_sp} - $tbl->{real_pages}, $tbl-
>{real_bytes} / ($tbl->{real_pages} * $PG_SIZE) * 100; foreach my $i ( sort keys %{$idx_stats{$t}} ) { my $idx =
$idx_stats{$t}->{$i}; printf $row_fmt, $t, $i, $idx->{est_sp}, $idx->{est_sp} - $idx->{real_pages}, $idx-
>{real_bytes} / ($idx->{real_pages} * $PG_SIZE) * 100; } } ```
The output of the above Perl script, when run against the sample shown in the previously mentioned blog
post, will appear as follows:
Expected output
{.text .no-copy}
TABLE INDEX TOT_PAGES FREE_PAGES PCT_FULL
art.link_out104 832383 38561 86.8%
art.link_out104 PRIMARY 498304 49 91.9%
art.link_out104 domain_id 49600 6230 76.9%
art.link_out104 domain_id_2 26495 3339 89.1%
art.link_out104 from_message_id 28160 142 96.3%
art.link_out104 from_site_id 38848 4874 79.4%
art.link_out104 revert_domain 153984 19276 71.4%
art.link_out104 site_message 36992 4651 83.4%
The columns are the table and index, followed by the total number of pages in that index, the number of
pages not actually occupied by data, and the number of bytes of real data as a percentage of the total size
of the pages of real data. The first line in the above output, in which the INDEX column is empty, is a
summary of the entire table.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
Note
FLUSH TABLES WITH READ LOCK does not prevent inserting rows into the log tables.
To ensure consistent backups, use the FLUSH TABLES WITH READ LOCK option before taking a non-InnoDB file
backup. The option does not affect long-running queries.
Long-running queries with FLUSH TABLES WITH READ LOCK enabled can leave the server in a read-only mode
until the queries finish. Killing the FLUSH TABLES WITH READ LOCK does not help if the database is in either the
Waiting for table flush or Waiting for master to send event state. To return to normal operation, you must
kill any long-running queries.
Note
All described in this section has no effect when backup locks are used. Percona XtraBackup will use Backup locks
where available as a lightweight alternative to FLUSH TABLES WITH READ
LOCK . This feature is available in Percona Server for MySQL 5.6+. Percona XtraBackup uses this automatically to
copy non-InnoDB data to avoid blocking DML queries that modify InnoDB tables.
In order to prevent this from happening two things have been implemented:
• xtrabackup kills all queries or only the SELECT queries which prevent the global lock from being
acquired
You should issue a global lock when no long queries are running. Waiting to issue the global lock for
extended period of time is not a good method. The wait can extend the time needed for backup to take
place. The –ftwrl-wait-timeout option can limit the waiting time. If it cannot issue the lock during this time,
xtrabackup stops the option, exits with an error message, and backup is not be taken.
The default value for this option is zero (0) value which turns off the option.
Another possibility is to specify the type of query to wait on. In this case --ftwrl-wait-query-type . Possible
values are all and update . When all is used xtrabackup will wait for all long running queries (execution
time longer than allowed by --ftwrl-wait-threshold ) to finish before running the FLUSH TABLES WITH READ
LOCK . When update is used xtrabackup will wait on UPDATE/ALTER/REPLACE/INSERT queries to finish.
The time needed for a specific query to complete is hard to predict. We assume that the long-running
queries will not finish in a timely manner. Other queries which run for a short time finish quickly. xtrabackup
uses the value of –ftwrl-wait-threshold option to specify the long-running queries and will block a global
lock. In order to use this option xtrabackup user should have PROCESS and SUPER privileges.
The second option is to kill all the queries which prevent from acquiring the global lock. In this case, all
queries which run longer than FLUSH TABLES WITH
READ LOCK are potential blockers. Although all queries can be killed, additional time can be specified for the
short running queries to finish using the --kill-long-queries-timeout option. This option specifies the time
for queries to complete, after the value is reached, all the running queries will be killed. The default value is
zero, which turns this feature off.
The --kill-long-query-type option can be used to specify all or only SELECT queries that are preventing
global lock from being acquired. In order to use this option xtrabackup user should have PROCESS and SUPER
privileges.
• --ftwrl-wait-timeout (seconds) - how long to wait for a good moment. Default is 0, not to wait.
• --ftwrl-wait-query-type - which long queries should be finished before FLUSH TABLES WITH READ LOCK is
run. Default is all.
• --ftwrl-wait-threshold (seconds) - how long query should be running before we consider it long
running and potential blocker of global lock.
• --kill-long-queries-timeout (seconds) - how many time we give for queries to complete after FLUSH
TABLES WITH READ LOCK is issued before start to kill. Default if 0 , not to kill.
Example
Running the xtrabackup with the following options will cause xtrabackup to spend no longer than 3 minutes
waiting for all queries older than 40 seconds to complete.
After FLUSH TABLES WITH READ LOCK is issued, xtrabackup will wait for 20 seconds for lock to be acquired. If
lock is still not acquired after 20 seconds, it will kill all queries which are running longer that the
FLUSH TABLES WITH
READ LOCK .
Contact us
For paid support and managed or consulting services , contact Percona Sales.
Percona XtraBackup has also implemented --lock-ddl-per-table , which blocks DDL statements by using
metadata locks (MDL).
The following procedures describe a simplified backup operation when using --lock-ddl-per-table :
1. Parse and copy all redo logs after the checkpoint mark
2. Fork a dedicated thread to continue following new redo log entries
3. List the tablespaces required to copy
4. Iterate through the list. The following steps occur with each listed tablespace:
5. Query INFORMATION_SCHEMA.INNODB_TABLES to find which tables belong to the tablespace ID and take an
MDL on the underlying table or tables in case there is a shared tablespace.
6. Copy the tablespace .ibd files.
The backup process may encounter a redo log event, generated by the bulk load operations, which notifies
backup tools that data file changes have been omitted from the redo log. This event is an MLOG_INDEX_LOAD .
If this event is found by the redo follow thread, the backup continues and assumes the backup is safe
because the MDL protects tablespaces already copied and the MLOG_INDEX_LOAD event is for a tablespace
that is not copied.
These assumptions may not be correct and may lead to inconsistent backups.
Implemented in Percona XtraBackup version 8.0.22-15.0, the --lock-ddl-per-table has been redesigned to
minimize inconsistent backups.
• Scan the redo logs. An MLOG_INDEX_LOAD event may be recorded if a CREATE INDEX statement has
occurred before the backup starts. At this time, the backup process is safe and can parse and accept
the event.
• After the first scan has completed, the follow redo log thread is initiated. This thread stops the backup
process if an MLOG_INDEX_LOAD event is found.
• Gather the tablespace list to copy
• If the .ibd file belongs to a temporary table, the SELECT query is skipped.
• For a FullText Index, an MDL is acquired on the base table.
• A SELECT query that acquires an MDL does not retrieve any data.
Warning
The lock-ddl-per-table variable is deprecated in Percona Server for MySQL 8.0. Use --lock-ddl instead of this
variable.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
To create an incremental backup with page tracking, Percona XtraBackup uses the MySQL mysqlbackup
component. This component provides a list of pages modified since the last backup, and Percona
XtraBackup copies only those pages. This operation removes the need to scan the pages in the database. If
the majority of pages have not been modified, the page tracking feature can improve the speed of
incremental backups.
{.bash data-prompt="$"}
$ SELECT COUNT(1) FROM mysql.component WHERE component_urn='file://component_mysqlbackup';
You can enable the page tracking functionality for the full and incremental backups with the --page-
tracking option.
• Resets page tracking to the start of the backup. This reset allows the next incremental backup to use
page tracking.
• Allows the use of page tracking for an incremental backup if the page tracking data is available from
the backup’s start checkpoint LSN.
Percona XtraBackup processes a list of all the tracked pages in memory. If Percona XtraBackup does not
have enough available memory to process this list, the process throws an error and exits. For example, if an
incremental backup uses 200GB, Percona XtraBackup can use an additional 100MB of memory to process
and store the page tracking data.
The examples of creating full and incremental backups using the --page-tracking option:
Full backup
{.bash data-prompt="$"}
$ xtrabackup --backup --target-dir=$FULL_BACK --page-tracking
Incremental backup
{.bash data-prompt="$"}
$ xtrabackup --backup --target-dir=$INC_BACKUP
--incremental-basedir=$FULL_BACKUP --page-tracking
After enabling the functionality, the next incremental backup finds changed pages using page tracking.
The first full backup using page tracking, Percona XtraBackup may have a delay. The following is an
example of the message:
Expected output
{.text .no-copy}
xtrabackup: pagetracking: Sleeping for 1 second, waiting for checkpoint lsn 17852922 /
to reach to page tracking start lsn 21353759
Enable page tracking before creating the first backup to avoid this delay. This method ensures that the
page tracking log sequence number (LSN) is higher than the checkpoint LSN of the server.
After the mysqlbackup component is loaded and active on the server, you can start page tracking manually
with the following option:
$ SELECT mysqlbackup_page_track_set(true);
Check the LSN value starting from which changed pages are tracked with the following option:
$ SELECT mysqlbackup_page_track_get_start_lsn();
$ SELECT mysqlbackup_page_track_set(false);
When you start page tracking, it creates a file under the server’s datadir to collect data about changed
pages. This file grows until you stop the page tracking. If you stop the server and then restart it, page
tracking creates a new file but also keeps the old one. The old file continues to grow until you stop the page
tracking explicitly.
If you purge the page tracking data, you should create a full backup afterward. To purge the page tracking
data, do the following steps:
$ SELECT mysqlbackup_page_track_set(false);
$ SELECT mysqlbackup_page_track_purge_up_to(9223372036854775807);
/* Specify the LSN up to which you want to purge page tracking data. /
9223372036854775807 is the highest possible LSN which purges all page tracking files.*/
$ SELECT mysqlbackup_page_track_set(true);
If the index is built in place using an exclusive algorithm and then is added to a table after the last LSN
checkpoint, you may generate a bad incremental backup using page tracking. For more details see
PS-8032.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
Note
In a Bash shell, the $? parameter returns the exit code from the last binary. If you use pipes, the ${PIPESTATUS[x]}
array parameter returns the exit code for each binary in the pipe string.
The xbcloud binary stores each chunk as a separate object with a name backup_name/database/
table.ibd.NNNNNNNNNNNNNNNNNNNN , where NNN... is a 0-padded serial number of chunk within a file. Size of
chunk produced by xtrabackup and xbstream changed to 10M.
To adjust the chunk size use --read-buffer-size . To adjust the chunk size for encrypted files, use --read-
buffer-size and --encrypt-chunk-size .
xbcloud has three essential operations: put, get, and delete. With these operations, backups are created,
stored, retrieved, restored, and deleted. xbcloud operations clearly map to similar operations within the AWS
Amazon S3 API.
The Exponential Backoff feature was implemented in Percona XtraBackup 8.0.26-18. Suppose a chunk fails to
upload or download. In that case, this feature adds an exponential backoff, or sleep, time and then retries
the upload or download, which increases the chances of completing a backup or a restore operation.
Important
To prevent intermittent backup failures, update the curl utility in Debian 10.
• OpenStack Object Storage (Swift) - see Using the xbcloud Binary with Swift
• Amazon Simple Storage (S3) - see Using xbcloud Binary with Amazon S3
• Azure Cloud Storage - see Using the xbcloud binary with Microsoft Azure Cloud Storage
• Google Cloud Storage (gcs) - see Using the xbcloud with Google Cloud Storage
• MinIO - see Using the xbcloud Binary with MinIO
In addition to OpenStack Object Storage (Swift), which has been the only option for storing backups in a
cloud storage until Percona XtraBackup 2.4.14, xbcloud supports Amazon S3, MinIO, and Google Cloud
Storage. Other Amazon S3-compatible storages, such as Wasabi or Digital Ocean Spaces, are also
supported.
See also
MinIO
Wasabi
11.2.2 Usage
An incremental backup only includes the changes since the last backup. The last backup can be either a full
or incremental backup.
To prepare an incremental backup, you must first download the full backup with the following command:
After the full backup has been prepared, download the incremental backup:
You do not need the full backup to restore only a specific database. You can specify only the tables to be
restored:
Each storage type has mandatory parameters that you can supply on the command line, in a configuration
file, and via environment variables.
Configuration files
The parameters the values of which do not change frequently can be stored in my.cnf or in a custom
configuration file. The following example is a template of configuration options under the [xbcloud] group:
[xbcloud]
storage=s3
s3-endpoint=http://localhost:9000/
s3-access-key=minio
s3-secret-key=minio123
s3-bucket=backupsx
s3-bucket-lookup=path
s3-api-version=4
Note
If you explicitly use a parameter on the command line and in a configuration file, xbcloud uses the value provided
on the command line.
Environment variables
If you explicitly use a parameter on the command line, in a configuration file, and the corresponding
environment variable contains a value, xbcloud uses the value provided on the command line or in the
configuration file.
Shortcuts
For all operations (put, get, and delete), you can use a shortcut to specify the storage type, bucket name,
and backup name as one parameter instead of using three distinct parameters (–storage, –s3-bucket, and
backup name per se).
Note
In this example s3 refers to a storage type, operator-testing is a bucket name, and bak22 is the backup name.
{.bash data-prompt="$"}
$ xbcloud get s3://operator-testing/bak22 ...
{.bash data-prompt="$"}
$ xbcloud get --storage=s3 --s3-bucket=operator-testing bak22 ...
You can supply the mandatory parameters on the command line, configuration files, and in environment
variables.
Additional parameters
xbcloud accepts additional parameters that you can use with any storage type. The --md5 parameter
computes the MD5 hash value of the backup chunks. The result is stored in files that following the
backup_name.md5 pattern.
You may use the --header parameter to pass an additional HTTP header with the server side encryption
while specifying a customer key.
The --header parameter is also useful to set the access control list (ACL) permissions:
--header="x-amz-acl: bucket-owner-full-control
First, you need to make the full backup on which the incremental one is going to be based:
--swift-tenant=admin --swift-password=xoxoxoxo \
--swift-auth-url=http://127.0.0.1:35357/ --parallel=10 \
full_backup | xbstream -xv -C /storage/downloaded_full
After the full backup has been prepared you can download the incremental backup:
Once the incremental backup has been downloaded you can prepare it by running:
If you do not want to download the entire backup to restore the specific database you can specify only the
tables you want to restore:
Contact us
For paid support and managed or consulting services , contact Percona Sales.
The following example shows how to make a full backup and upload it to Swift.
The following OpenStack environment variables are also recognized and mapped automatically to the
corresponding swift parameters ( --storage=swift ):
• OS_AUTH_URL
• OS_TENANT_NAME
• OS_TENANT_ID
• OS_USERNAME
• OS_PASSWORD
• OS_USER_DOMAIN
• OS_USER_DOMAIN_ID
• OS_PROJECT_DOMAIN
• OS_PROJECT_DOMAIN_ID
• OS_REGION_NAME
• OS_STORAGE_URL
• OS_CACERT
The following example shows how to fetch and restore the backup from Swift:
--storage(=[swift|s3|google])
Cloud storage option. xbcloud supports Swift, MinIO, and AWS S3. The default value is swift .
--swift-auth-url()
--swift-storage-url()
xbcloud will try to get object-store URL for given region (if any specified) from the keystone response. One
can override that URL by passing –swift-storage-url=URL argument.
--swift-user()
--swift-key()
--swift-container()
--parallel(=N)
--cacert()
--insecure()
Swift specification describes several authentication options. xbcloud can authenticate against keystone
with API version 2 and 3.
--swift-auth-version()
Specifies the swift authentication version. Possible values are: 1.0 - TempAuth, 2.0 - Keystone v2.0, and 3
- Keystone v3. Default value is 1.0 .
--swift-tenant()
--swift-tenant-id()
--swift-region()
--swift-password()
--swift-user-id()
--swift-project()
--swift-project-id()
--swift-domain()
--swift-domain-id()
Contact us
For paid support and managed or consulting services , contact Percona Sales.
Option Details
–s3-region Use to specify the AWS region. The default value is us-east-1
–s3-api- Select the signing algorithm. The default value is AUTO. In this case, xbcloud will probe.
version =
–s3-storage- Specify the S3 storage class. The default storage class depends on the provider. The
class= name options are the following:
• STANDARD
• STANDARD_IA
• GLACIER
NOTE If you use the GLACIER storage class, the object must be restored to S3 before
restoring the backup. Also supports using custom S3 implementations such as MinIO or
CephRadosGW.
Following the principle of "least-privilege", these are the minimum bucket permissions needed for xbcloud to
write backups to S3: LIST/PUT/GET/DELETE.
The following example shows the policy definition for writing to the xbcloud-testing bucket on the AWS S3
storage.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": "arn:aws:s3:::xbcloud-testing"
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:PutObjectAcl",
"s3:GetObject",
"s3:GetObjectAcl",
"s3:DeleteObject"
],
"Resource": "arn:aws:s3:::xbcloud-testing/*"
}
]
}
The following environment variables are recognized. xbcloud maps them automatically to corresponding
parameters applicable to the selected storage.
• AWS_CA_BUNDLE
Contact us
For paid support and managed or consulting services , contact Percona Sales.
XtraBackup 8.0.31-24 add the ability to use the IAM instance profile when running xbcloud from an EC2
instance.
• Who am I?
• What can I do?
A role defines "what can I do." A role provides a method to define a collection of permissions. Roles are
assigned to users, services and EC2 instances, the "who am I" element.
The IAM instance profile is the "who" for an EC2 instance and assumes the IAM role, which has permissions.
The instance profile has the same name as the IAM role.
An IAM instance profile does not need the --s3-secret-key nor the --s3-access-key if they are running
xbcloud from an Amazon EC2 instance. To configure or attach an instance metadata to an EC2 instance,
see How can I grant my Amazon EC2 instance access to an Amazon S3 bucket.
Expected output
text
221121 13:16:26 Using instance metadata for access and secret key
221121 13:16:26 xbcloud: Successfully connected.
An important consideration is that the instance metadata has a time to live (TTL) of 6 hours. A backup that
takes more than that time contains Expired token errors. Use Exponential Backoff to retry the upload after
fetching new keys from the instance metadata.
text
221121 13:04:52 xbcloud: S3 error message: The provided token has expired.
221121 13:04:52 xbcloud: Sleeping for 2384 ms before retrying test/mysql.ibd.00000000000000000002 [1]
221121 13:04:55 xbcloud: S3 error message: The provided token has expired.
221121 13:04:55 xbcloud: Sleeping for 2887 ms before retrying test/mysql.ibd.00000000000000000003 [1]
221121 13:04:58 xbcloud: S3 error message: The provided token has expired.
221121 13:04:58 xbcloud: Sleeping for 2778 ms before retrying test/undo_002.00000000000000000000 [1]
221121 13:05:00 xbcloud: S3 error message: The provided token has expired.
221121 13:05:00 xbcloud: Sleeping for 2916 ms before retrying test/undo_002.00000000000000000001 [1]
221121 13:05:03 xbcloud: S3 error message: The provided token has expired.
221121 13:05:03 xbcloud: Sleeping for 2794 ms before retrying test/undo_002.00000000000000000002 [1]
221121 13:05:06 xbcloud: S3 error message: The provided token has expired.
221121 13:05:06 xbcloud: Sleeping for 2336 ms before retrying test/undo_001.00000000000000000000 [1]
221121 13:05:09 xbcloud: successfully uploaded chunk: test/mysql.ibd.00000000000000000002, size: 5242923
221121 13:05:09 xbcloud: successfully uploaded chunk: test/mysql.ibd.00000000000000000003, size: 23
221121 13:05:09 xbcloud: successfully uploaded chunk: test/undo_002.00000000000000000000, size: 10485802
221121 13:05:09 xbcloud: successfully uploaded chunk: test/undo_002.00000000000000000001, size: 6291498
221121 13:05:09 xbcloud: successfully uploaded chunk: test/undo_002.00000000000000000002, size: 22
221121 13:05:09 xbcloud: successfully uploaded chunk: test/undo_001.00000000000000000000, size: 10485802
221121 13:05:10 xbcloud: successfully uploaded chunk: test/undo_001.00000000000000000001, size: 6291498
221121 13:05:10 xbcloud: successfully uploaded chunk: test/undo_001.00000000000000000002, size: 22
. . .
221121 13:05:18 xbcloud: successfully uploaded chunk: test/xtrabackup_tablespaces.00000000000000000001, size:
36
221121 13:05:19 xbcloud: Upload completed.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
The support for Google Cloud Storage is implemented using the interoperability mode. This mode was
especially designed to interact with cloud services compatible with Amazon S3.
See also
The following options are available when using Google Cloud Storage:
• –google-access-key =
• –google-secret-key =
• –google-bucket =
• –google-storage-class=name
Note
• STANDARD
• NEARLINE
• COLDLINE
• ARCHIVE
See also
Google storage classes - the default Google storage class depends on the storage class of the bucket
Contact us
For paid support and managed or consulting services , contact Percona Sales.
Exponential backoff increases the chances for the completion of a backup or a restore operation. For
example, a chunk upload or download may fail if you have an unstable network connection or other network
issues. This feature adds an exponential backoff, or sleep, time and then retries the upload or download.
When a chunk upload or download operation fails, xbcloud checks the reason for the failure. This failure can
be a CURL error or an HTTP error, or a client-specific error. If the error is listed in the Retriable errors list,
xbcloud pauses for a calculated time before retrying the operation until that time reaches the
--max-backoff value.
The operation is retried until the --max-retries value is reached. If the chunk operation fails on the last retry,
xbcloud aborts the process.
• –max-retries = 10
You can adjust the number of retries by adding the --max-retries parameter and adjust the maximum
length of time between retries by adding the --max-backoff parameter to an xbcloud command.
Since xbcloud does multiple asynchronous requests in parallel, a calculated value, measured in
milliseconds, is used for max-backoff . This algorithm calculates how many milliseconds to sleep before the
next retry. A number generated is based on the combining the power of two (2), the number of retries
already attempted and adds a random number between 1 and 1000. This number avoids network
congestion if multiple chunks have the same backoff value. If the default values are used, the final retry
attempt to be approximately 17 minutes after the first try. The number is no longer calculated when the
milliseconds reach the --max-backoff setting. At that point, the retries continue by using the --max-backoff
setting until the max-retries parameter is reached.
• CURLE_GOT_NOTHING
• CURLE_OPERATION_TIMEOUT
• CURLE_RECV_ERROR
• CURLE_SEND_ERROR
• CURLE_SEND_FAIL_REWIND
• CURLE_PARTIAL_FILE
• CURLE_SSL_CONNECT_ERROR
• 503
• 500
• 504
• 408
Each cloud provider may return a different CURL error or an HTTP error, depending on the issue. Add new
errors by setting the following variables --curl-retriable-errors or --http-retriable-errors on the
command line or in my.cnf or in a custom configuration file under the [xbcloud] section. You can add
multiple errors using a comma-separated list of codes.
The error handling is enhanced when using the --verbose output. This output specifies which error caused
xbcloud to fail and what parameter a user must add to retry on this error.
Expected output
{.text .no-copy}
210701 14:34:23 /work/pxb/ins/8.0/bin/xbcloud: Operation failed. Error: Server returned nothing (no headers,
no data)
210701 14:34:23 /work/pxb/ins/8.0/bin/xbcloud: Curl error (52) Server returned nothing (no headers, no data)
is not configured as retriable. You can allow it by adding --curl-retriable-errors=52 parameter
11.8.2 Example
The following example adjusts the maximum number of retries and the maximum time between retries.
• The chunk xtrabackup_logfile.00000000000000000006 fails to upload the first time and sleeps for 2384
milliseconds.
• The same chunk fails for the second time and the sleep time is increased to 4387 milliseconds.
• The same chunk fails for the third time and the sleep time is increased to 8691 milliseconds.
• The same chunk fails for the fourth time. The max-backoff parameter has been reached. All retries sleep
the same amount of time after reaching the parameter.
{.text .no-copy}
210702 10:07:05 /work/pxb/ins/8.0/bin/xbcloud: Operation failed. Error: Server returned nothing (no headers,
no data)
210702 10:07:05 /work/pxb/ins/8.0/bin/xbcloud: Sleeping for 2384 ms before retrying backup3/xtrabackup_logfile.
00000000000000000006
. . .
210702 10:07:23 /work/pxb/ins/8.0/bin/xbcloud: Operation failed. Error: Server returned nothing (no headers,
no data)
210702 10:07:23 /work/pxb/ins/8.0/bin/xbcloud: Sleeping for 4387 ms before retrying backup3/xtrabackup_logfile.
00000000000000000006
. . .
210702 10:07:52 /work/pxb/ins/8.0/bin/xbcloud: Operation failed. Error: Failed sending data to the peer
210702 10:07:52 /work/pxb/ins/8.0/bin/xbcloud: Sleeping for 8691 ms before retrying backup3/xtrabackup_logfile.
00000000000000000006
. . .
210702 10:08:47 /work/pxb/ins/8.0/bin/xbcloud: Operation failed. Error: Failed sending data to the peer
210702 10:08:47 /work/pxb/ins/8.0/bin/xbcloud: Sleeping for 10000 ms before retrying backup3/
xtrabackup_logfile.00000000000000000006
. . .
210702 10:10:12 /work/pxb/ins/8.0/bin/xbcloud: successfully uploaded chunk: backup3/xtrabackup_logfile.
00000000000000000006, size: 8388660
Contact us
For paid support and managed or consulting services , contact Percona Sales.
11.9 Use the xbcloud binary with Microsoft Azure Cloud Storage
Implemented in Percona XtraBackup 8.0.27-19, the xbcloud binary adds support for the Microsoft Azure
Cloud Storage using the REST API.
11.9.1 Options
The following are the options, environment variables, and descriptions for uploading a backup to Azure
using the REST API. The environment variables are recognized by xbcloud, which maps them automatically
to the corresponding parameters:
Test your Azure applications with the Azurite open-source emulator. For testing purposes, the xbcloud
binary adds the --azure-development-storage option that uses the default access_key and storage account
of azurite and testcontainer for the container. You can overwrite these options, if needed.
11.9.2 Usage
All the available options for xbcloud, such as parallel, max-retries, and others, can be used. For more
information, see the xbcloud Binary.
11.9.3 Examples
Contact us
For paid support and managed or consulting services , contact Percona Sales.
12.1.2 How-tos
• How to set up a replica for replication in 6 simple steps with Percona XtraBackup
Most of the time, the context will make the recipe or tutorial understandable. To assure that, a list of the
assumptions, names and “things” that will appear in this section is given. At the beginning of each recipe or
tutorial they will be specified in order to make it quicker and more practical.
HOST
A system with a MySQL-based server installed, configured and running. We will assume the following about
this system:
• the MySQL server is able to communicate with others by the standard TCP/IP port;
• an SSH server is installed and configured - see here if it is not;
• you have a user account in the system with the appropriate permissions and
• you have a MySQL’s user account with appropriate Connection and Privileges Needed.
USER
An user account in the system with shell access and appropriate permissions for the task. A guide for
checking them is here.
DB-USER
An user account in the database server with appropriate privileges for the task. A guide for checking them is
here.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
Percona XtraBackup for MySQL Databases enables MySQL backups without blocking user queries. Percona
XtraBackup is ideal for companies with large data sets and mission-critical applications that cannot
tolerate long periods of downtime. Offered free as an open source solution, Percona XtraBackup drives down
backup costs while providing unique features for MySQL backups.
This release includes improvements and bug fixes. Percona has implemented a two-stage release process
for each version. The first release primarily ensures compatibility with the latest MySQL version, to help those
customers who need an updated version of Percona XtraBackup as soon as possible. The second release
contains additional bug fixes and any improvements or new features.
This version adds the --estimate-memory parameter to enable/disable the Smart memory estimation feature
during the backup phase.
13.2.2 Improvements
PXB-2979: The Smart memory estimation feature parses metadata from the redo log to estimate the
required memory to prepare a backup. On write-intensive workloads, this behavior caused the delay of the
redo follow thread.
PXB-2980: Adds the --estimate-memory parameter to enable/disable the Smart memory estimation feature
during the backup phase.
PXB-2264: When a table with the next auto incremental value 2 was exported using Percona XtraBackup,
the table got the next auto incremental value 1 instead of 2 .
PXB-2954: During prepare phase Percona XtraBackup uses Serialized Dictionary information (SDI) to create
the tabels's metadata. If there was an orphan table (leftover from a upgrade) without SDI information,
Percona XtraBackup failed to prepare a backup.
PXB-2970: Removed unnecessary check for redo log files at the end of backup in case the redo archive was
enabled.
PXB-2972: An error after the redo apply phase caused an assertion failure instead of a graceful exit.
PXB-2998: Due to a server bug in the upgrade, the server leaves the dictionary tables in a temporary
schema instead of the mysql schema. Percona Xtrabackup can now successfully prepare these backup
directories.
PXB-2999: Removed the warning message that MLOG_INDEX_LOAD redo log record was found.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
Percona XtraBackup for MySQL Databases enables MySQL backups without blocking user queries. Percona
XtraBackup is ideal for companies with large data sets and mission-critical applications that cannot
tolerate long periods of downtime. Offered free as an open source solution, Percona XtraBackup drives down
backup costs while providing unique features for MySQL backups.
This release merges the MySQL 8.0.32 code base. Percona has implemented a two-stage release process for
each version. The first release primarily ensures compatibility with the latest MySQL version, to help those
customers who need an updated version of Percona XtraBackup as soon as possible. The second release
contains additional bug fixes and any improvements or new features.
Starting with version 8.0.32-25, Percona XtraBackup allows instant columns in the backup for MySQL 8.0.32.
This limitation still exists for backing up MySQL 8.0.31 and lower versions.
This limitation does not exist for backing up any version of Percona Server for MySQL.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
Percona XtraBackup for MySQL Databases enables MySQL backups without blocking user queries. Percona
XtraBackup is ideal for companies with large data sets and mission-critical applications that cannot
tolerate long periods of downtime. Offered free as an open source solution, Percona XtraBackup drives down
backup costs while providing unique features for MySQL backups.
This release adds the ability to use an IAM instance profile when running xbcloud from an EC2 instance. An
IAM instance profile does not need the --s3-secret-key nor the --s3-access-key but the profile has a six-
hour lifespan.
PXB-2856: Use IAM instance profile when running xbcloud from an EC2 instance.
PXB-2928: XtraBackup exits with signal 11 when taking a backup using page tracking.
PXB-2925: The PXB start_lsn, during the prepare phase, did not match the redo file start LSN when a wait was
added in the backup phase after calling recv_find_max_checkpoint and before copying the checkpoint
header.
PXB-2971: XtraBackup copied local manifest files which caused node failures when the files overrode
settings on another node.
Starting with Percona XtraBackup 8.0.31-24, using qpress/QuickLZ to compress backups is deprecated and
may be removed in future versions. We recommend using either the LZ4 or the Zstandard ( ZSTD )
compression algorithms.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
Percona XtraBackup for MySQL Databases enables MySQL backups without blocking user queries. Percona
XtraBackup is ideal for companies with large data sets and mission-critical applications that cannot
tolerate long periods of downtime. Offered free as an open source solution, Percona XtraBackup drives down
backup costs while providing unique features for MySQL backups.
For paid support, managed services or consulting services, contact Percona Sales.
The Take an incremental backup using page tracking feature is Generally Available (GA).
The following features and improvements introduced in Percona XtraBackup 8.0.30-23 are available only in
tech preview:
• Percona XtraBackup adds support for Zstandard (ZSTD) compression algorithm. ZSTD is a fast lossless
compression algorithm, targeting real-time compression scenarios at zlib-level and better
compression ratios.
• Percona XtraBackup implements the Smart memory estimation feature to compute the memory
required to --prepare a backup. Percona XtraBackup has extended the crash recovery logic to extract
the formula used to allocate memory. Now, in the backup phase, while copying redo log entries,
Percona XtraBackup computes the required memory for --prepare . Percona XtraBackup also takes into
consideration the number of InnoDB pages to be fetched from the disk. Percona XtraBackup checks the
server's available free memory and uses that memory up to the limit specified in the
--use-free-memory-pct option to run --prepare .
13.5.3 Improvements
PXB-2681: Percona XtraBackup got stuck during incremental backup prepare of encrypted tables.
PXB-2517: Check if files-from is successfully closed before rsync (Thanks to Garen Chan for the patch.)
PXB-2702: Backup failed when undo truncation log is present (Thanks to Dmitry Smal for his contribution to
the initial patch.)
PXB-2810: Percona XtraBackup crashed when AIO initialization failed (Thanks to Yan Huang for the patch.)
PXB-2874: When Percona XtraBackup generated a new master key on --copy/move-back , the master key id
was reset back to 1.
PXB-2809: wsrep_sync_wait<>0 caused Lock wait timeout exceeded; try restarting transaction error
(Thanks to Frank Well for providing the initial patch.)
Contact us
For paid support and managed or consulting services , contact Percona Sales.
Percona XtraBackup 8.0.29-22 adds new redo log types to support the changes in the INSTANT algorithm
behavior. MySQL 8.0.29 extended the support for ALGORITHM=INSTANT to allow columns to be added to any
position in a table and column drops. Older versions of Percona XtraBackup are incompatible with MySQL
8.0.29 because of this new functionality.
Note
MySQL 8.0.29 extended the support of the ALTER TABLE … DROP COLUMNS statement to use ALGORITHM=INSTANT . These
operations only modify the metadata in the data directory and do not affect the physical structure. The
ALGORITHM=INSTANT is the default setting for supported DDL operations. Prior to MySQL 8.0.29, an ALGORITHM=INSTANT
column could only be added as the last table column. As of MySQL 8.0.29, the column can be added to any table
position. The ability to add or drop a column in any position increases the complexity of crash recovery since the
redo log does not have the logical positions from the data dictionary. The redo log contains the physical order for
the table.
Percona Server for MySQL 8.0.29-21 contains fixes for these issues. Percona XtraBackup 8.0.29 can backup and
restore tables that used the INSTANT algorithm in ADD/DROP in Percona Server for MySQL. However, MySQL 8.0.29, if
the server contains tables modified by INSTANT ADD/DROP, is unsafe for backups at this time. Percona XtraBackup
detects the tables modified by INSTANT ADD/DROP and generates an error. This error lists the affected tables and
provides instructions to convert the modified tables to regular tables.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
Percona XtraBackup adds support for the Amazon Key Management Service (KMS) component.
• PXB-2721: Implements support for the Amazon Key Management Service component in Percona
XtraBackup.
• PXB-2761: Fix for a compilation error in GCC 11. (Thanks to user tcoyvwac for providing the patch to fix
this issue.)
• PXB-2422: The extraction of files failed when a file was located in another directory.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
Percona XtraBackup for MySQL Databases enables MySQL backups without blocking user queries. Percona
XtraBackup is ideal for companies with large data sets and mission-critical applications that cannot
tolerate long periods of downtime. Offered free as an open source solution, Percona XtraBackup drives down
backup costs while providing unique features for MySQL backups.
The log statements structure in Percona XtraBackup has been improved. Now, the error logs have a
standard structure. The uniformity of the headers makes it easier to follow an operation’s progress or review
the log to diagnose issues. The improved log structure is displayed in the backup, prepare, move-back/
copy-back error logs.
• Context - the name of the module that issued the log statement, such as XtraBackup, InnoDB, or
Server.
• PXB-2716: Fix for a compilation error on systems without fallocate. (Thanks to user Bo98 for providing the
patch to fix this issue.)
• PXB-2506: During copy back and move back operations, added a check if the backup is fully prepared.
Percona XtraBackup throws an error if the backup is not prepared or was prepared with –apply-log-
only.
• PXB-2662: Percona XtraBackup exited during prepare when creating an incremental backup using
page tracking.
• PXB-2714: Fix for a memory leak on the keyring component.
• PXB-2697: Undefined symbols in macOS resulted in an error when linked with Percona XtraBackup
8.0.27.
• PXB-2722: Fix for when via command line, a password, passed using the -p option, was written into the
backup tool_command in xtrabackup_info.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
• Installation
Installing Percona XtraBackup
Percona XtraBackup enables MySQL backups without blocking user queries, making it ideal for companies
with large data sets and mission-critical applications that cannot tolerate long periods of downtime.
Offered free as an open source solution, it drives down backup costs while providing unique features for
MySQL backups.
The following list contains a number of the bug fixes for MySQL 8.0.27, provided by Oracle, and included in
Percona Server for MySQL:
• The default_authentication_plugin is deprecated. Support for this plugin may be removed in future
versions. Use the authentication_policy variable.
• The binary operator is deprecated. Support for this operator may be removed in future versions. Use
CAST(... AS BINARY) .
• When a parent table updated or deleted a row, this operation initiated a cascading SET NULL operation
on the child table. On the child table, a virtual column value could be set to NULL instead of the value
derived from the base column value.
Find the full list of bug fixes and changes in the MySQL 8.0.27 Release Notes.
• PXB-1883: Added support for Microsoft Azure Cloud Storage in the xbcloud binary. (Thanks to Ivan for
reporting this issue)
13.9.3 Improvements
• PXB-1741: PS startup displays errors for databases excluded during partial backup
• PXB-2614: The xbstream binary was enhanced to support sparse files on the XFS file system.
• PXB-2608: Upgrade the Vault API to V2 (Thanks to Benedito Marques Magalhaes for reporting this issue)
• PXB-2543: Fix that adds IB_EXPORT_CFG_VERSION_V6 when exporting a .cfg file for a table (Thanks to
Peng Gao for reporting this issue)
• PXB-2465: Fix for querying the ps.log_status when group replication channels are items on that list.
XtraBackup skips the name if it matches either group_replication_applier or
group_replication_recovery .(Thanks to Galamb Gergő for reporting this issue)
• PXB-2625: The default version of CURL in Debian Buster failed to identify a TLS HTTP2 connection as
closed and attempted to reuse the connection. (Thanks to Johan Andersson for reporting this issue)
Contact us
For paid support and managed or consulting services , contact Percona Sales.
• Installation
Installing Percona XtraBackup
Percona XtraBackup enables MySQL backups without blocking user queries, making it ideal for companies
with large data sets and mission-critical applications that cannot tolerate long periods of downtime.
Offered free as an open source solution, it drives down backup costs while providing unique features for
MySQL backups.
13.10.1 Improvements
• PXB-2477: The xbcloud should retry on error and utilize incremental backoff (Thanks to Baptiste Mille-
Mathias for reporting this issue)
• PXB-2580: With the xbcloud binary, a chunk-upload on an SSL connect error to S3 was not retried
(Thanks to Tim Vaillancourt for providing the patch)
• PXB-2317: Remove the obsolete LOCKLESS binary log functionality since the
performance_schema.log_status table is now used to get the log information on all the storages without
locking them
• PXB-2101: The Prepare step fails due to duplicate Serialized Dictionary Information (SDI) (Thanks to
Fungo Wang for reporting this issue)
• PXB-1504: The FIND_GCRYPT macro is broken (Thanks to Maxim Bublis for reporting this issue)
• PXB-1961: The Prepare step of a full backup fails for encrypted tables with a generic error
• PXB-2585: XtraBackup fails to backup archive RocksDB Write-Ahead Log (WAL) files
Contact us
For paid support and managed or consulting services , contact Percona Sales.
• Installation
Installing Percona XtraBackup
Percona XtraBackup enables MySQL backups without blocking user queries, making it ideal for companies
with large data sets and mission-critical applications that cannot tolerate long periods of downtime.
Offered free as an open source solution, it drives down backup costs while providing unique features for
MySQL backups.
• PXB-2444 : The xbcloud binary fails to upload backup with enforcing SELinux mode
• PXB-2455 : XtraBackup prepare fails if the checkpoint LSN is greater than the last LSN
Contact us
For paid support and managed or consulting services , contact Percona Sales.
• Installation
Installing Percona XtraBackup
Percona XtraBackup enables MySQL backups without blocking user queries, making it ideal for companies
with large data sets and mission-critical applications that cannot tolerate long periods of downtime.
Offered free as an open source solution, it drives down backup costs while providing unique features for
MySQL backups.
13.12.1 Improvements
• PXB-2274: Correct restore processing when there are DML statements running during backup stage by
writing the last_checkpoint and LSN from ps.log_status instead of the redo log (Thanks to user Li Biao
for reporting this issue)
• PXB-2430: Correct build failure by removing the dependency of no-server-version-check with version-
check (Thanks to user agarner for reporting this issue)
• PXB-2415: Add build dependencies to correct Debian/Ubuntu packages in docker (Thanks to user Matt
Cole for reporting this issue)
• PXB-2429: Correct Backup failure for encrypted tablespace by skipping the encryption reset operation
during the redo scan phase
• PXB-2418: Remove PROTOBUF_LITE_LIBRARY - it is not used and is not needed
• PXB-2395: Update versions for xbstream and xbcrypt
• PXB-2357: Correct hang in backup with redo log archive by using block number instead of checksum
(checksum of redo file and archive file can be different)
• PXB-2180: Correct incremental prepare failure with logical redo by skipping the apply of logical redos
(MLOG_TABLE_DYNAMIC_META) during the incremental prepare (except the last prepare).
Contact us
For paid support and managed or consulting services , contact Percona Sales.
• Installation
Installing Percona XtraBackup
Percona XtraBackup enables MySQL backups without blocking user queries, making it ideal for companies
with large data sets and mission-critical applications that cannot tolerate long periods of downtime.
Offered free as an open source solution, it drives down backup costs while providing unique features for
MySQL backups.
NOTE: The naming structure for the releases has changed. See the aligning Percona XtraBackup versions
with Percona Server for MySQL blog post for more information.
• PXB-2112: xbcloud: support storage_class option with –storage=s3 (Thanks to user rluisr for reporting
this issue)
• PXB-2242: Prevent back up if Source system (DB) version is higher then current PXB version
13.13.2 Improvements
• PXB-2336: Modify to check to see if first block is an all-zero block and process accordingly
• PXB-2275: Modify backup processing to add validations if an encrypted table is created
• PXB-2272: Modify regexp to consider all #sql as temporary tables
Contact us
For paid support and managed or consulting services , contact Percona Sales.
• Installation
Installing Percona XtraBackup
Percona XtraBackup enables MySQL backups without blocking user queries, making it ideal for companies
with large data sets and mission-critical applications that cannot tolerate long periods of downtime.
Offered free as an open source solution, it drives down backup costs while providing unique features for
MySQL backups.
Percona XtraBackup 8.0.14 supports backup and restore processing for all versions of MySQL and has been
tested with the latest MySQL 8.0.21.
13.14.1 Improvements
• PXB-2252: Introduce debug option to print redo log records scanned and applied
• PXB-2114: Document when table is ignored by full backup it is not automatically ignored for incremental
backup
• PXB-2243: Correct processing when undo truncation happens between full backup and incremental
• PXB-2238: Provide binary tarball with shared libs and glibc suffix & minimal tarballs
• PXB-2202: Modify Xbcloud to display an error when xtrabackup fails to create a backup
Contact us
For paid support and managed or consulting services , contact Percona Sales.
• Installation
Installing Percona XtraBackup
Percona XtraBackup enables MySQL backups without blocking user queries, making it ideal for companies
with large data sets and mission-critical applications that cannot tolerate long periods of downtime.
Offered free as an open source solution, it drives down backup costs while providing unique features for
MySQL backups.
Percona XtraBackup 8.0.13 supports backup and restore processing for all versions of MySQL and has been
tested with the latest MySQL 8.0.20.
• PXB-2165: Modify xbcloud to store backups using s3 access key parameters if AWS access key env
variables are set
• PXB-2164: Modify xbcloud to return the error when the backup doesn’t exist in s3 bucket
• PXB-2127: Modify xbcloud to upload backups with empty database to min.io storage
• PXB-2198: Modify xbcloud delete to return the error when the backup doesn’t exist in s3 bucket
• PXB-2155: Correct corruption when redo logs are encrypted using xtrabackup –backup –compress=lz4
• PXB-2166: Modify xbcloud to check for bucket existence and return ok status when domain name can
be resolved
Contact us
For paid support and managed or consulting services , contact Percona Sales.
• Installation
Installing Percona XtraBackup
Percona XtraBackup enables MySQL backups without blocking user queries, making it ideal for companies
with large data sets and mission-critical applications that cannot tolerate long periods of downtime.
Offered free as an open source solution, it drives down backup costs while providing unique features for
MySQL backups.
Percona XtraBackup 8.0.12 now supports backup and restore processing for all versions of MySQL; previous
versions of Percona XtraBackup will not work with MySQL 8.0.20 and higher.
• PXB-2133: Correct prtype stored for DECIMAL columns in .cfg file by –export
• PXB-2179: Modify xtrabackup prepare to shut down keyring plugin when run with –generate-transition-
key
• PXB-2157: Modify xtrabackup –prepare to generate cfg files for partition tables when –export is used
Contact us
For paid support and managed or consulting services , contact Percona Sales.
Downloads are available from our download site and from apt and yum repositories.
Percona XtraBackup enables MySQL backups without blocking user queries, making it ideal for companies
with large data sets and mission-critical applications that cannot tolerate long periods of downtime.
Offered free as an open source solution, it drives down backup costs while providing unique features for
MySQL backups.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
Downloads are available from our download site and from apt and yum repositories.
Percona XtraBackup enables MySQL backups without blocking user queries, making it ideal for companies
with large data sets and mission-critical applications that cannot tolerate long periods of downtime.
Offered free as an open source solution, it drives down backup costs while providing unique features for
MySQL backups.
• PXB-2099: copy-back did not move undo files to the data directory.
• PXB-2107: If the undo tablespace was created in a directory which is a symbolic link to another directory
then the backup failed at backup stage.
• PXB-2118: Undo log file was renamed incorrectly to undo tablespace name after restore
Contact us
For paid support and managed or consulting services , contact Percona Sales.
Percona XtraBackup enables MySQL backups without blocking user queries, making it ideal for companies
with large data sets and mission-critical applications that cannot tolerate long periods of downtime.
Offered free as an open source solution, it drives down backup costs while providing unique features for
MySQL backups.
• Sometime between December 3rd and December 10th, a change was introduced in that caused an
incompatibility with our Percona XtraBackup xbcloud utility. Bug fixed PXB-1978.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
Percona XtraBackup enables MySQL backups without blocking user queries, making it ideal for companies
with large data sets and mission-critical applications that cannot tolerate long periods of downtime.
Offered free as an open source solution, it drives down backup costs while providing unique features for
MySQL backups.
• Support log archiving feature in PXB 8.0. More information in PXB-1912 and in the Redo Log section of
MySQL documentation
• For the MyRocks storage engine, support the creation of renewable checkpoints (controlled via --
rocksdb-checkpoint-max-age and --rocksdb-checkpoint-max-count ) to minimize the amount of binary
logs to apply after the backup was completed. Using renewable checkpoints, Percona XtraBackup only
copies the SST files that were created after the previous checkpoint. More information in PXB-1915.
• An encrypted table could not be restored when ADD INDEX or DROP INDEX commands had been run on
the table. Bug fixed PXB-1905
• In some cases xtrabackup --prepare could fail to decrypt a table but reported that the operation
completed ok. Bug fixed PXB-1936
• xtrabackup --move-back did not complete successfully when the encrypted binlog file. Bug fixed
PXB-1937.
• Percona XtraBackup could crash during the prepare stage when making an incremental
backup when a multi valued index was being added or dropped for JSON data. Bug fixed PXB-1913.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
Percona XtraBackup enables MySQL backups without blocking user queries, making it ideal for companies
with large data sets and mission-critical applications that cannot tolerate long periods of downtime.
Offered free as an open source solution, it drives down backup costs while providing unique features for
MySQL backups.
In release 8.0.7, Percona XtraBackup enables making backups of databases that contain the encrypted
system tablespace. Encrypted mysql tablespace is now also supported.
Percona XtraBackup 8.0.7 implements the support of the lz4 compression algorithm so that you could
make compressed backups using lz4 (–compress=lz4) in addition to the default quicklz method.
• Implemented the support of the lz4 compression algorithm. More information in PXB-1857.
• When the encrypted tablespaces feature was enabled, encrypted and compressed tables were not
usable on the joiner node (Percona XtraDB Cluster) via SST (State Snapshot Transfer) with the
xtrabackup-v2 method. Bug fixed PXB-1867.
• xbcloud did not update date related fields of the HTTP header when retrying a request. Bug fixed
PXB-1874.
• xbcloud did not retry to send the request after receiving the HTTP 408 error (request timeout). Bug fixed
PXB-1875.
• xtrabackup did not accept decimal fractions as values of the innodb_max_dirty_pages_pct option. Bug
fixed PXB-1807.
• If the user tried to merge an already prepared incremental backup, a misleading error was produced
without informing that incremental backups may not be used twice. Bug fixed PXB-1862.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
Percona XtraBackup enables MySQL backups without blocking user queries, making it ideal for companies
with large data sets and mission-critical applications that cannot tolerate long periods of downtime.
Offered free as an open source solution, it drives down backup costs while providing unique features for
MySQL backups.
In version 8.0.6, Percona XtraBackup introduces the support of the MyRocks storage engine with Percona
Server for MySQL version 8.0.15-6 or higher.
Percona XtraBackup 8.0.6 enables saving backups to an Amazon S3, MinIO, and Google Cloud Storage
(using interoperability mode) when using xbcloud. The following example demonstrates how to use an
Amazon S3 storage to make a full backup:
• The MyRocks storage engine is now supported with Percona XtraBackup. More information in PXB-1754.
• Amazon S3 is now supported in xbcloud. More information in PXB-1813.
• Percona XtraBackup could fail to restore the undo tablespace created during or before incremental
backup. Bug fixed PXB-1780.
• A backup could fail if log_bin_index was defined in my.cnf . Bug fixed PXB-1801.
• When the row format was changed during the backup, xtrabackup could crash during the incremental
prepare stage. Bug fixed PXB-1824.
• During the prepare phase, Percona XtraBackup could freeze and never finish execution. Bug fixed
PXB-1819.
• Percona XtraBackup could crash during the prepare stage when making a backup of a host running
MySQL Server v8.0.16. Bug fixed PXB-1839.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
Percona XtraBackup enables MySQL backups without blocking user queries, making it ideal for companies
with large data sets and mission-critical applications that cannot tolerate long periods of downtime.
Offered free as an open source solution, it drives down backup costs while providing unique features for
MySQL backups.
Percona XtraBackup 8.0.5 introduces the support of undo tablespaces created using the new syntax
( CREATE UNDO TABLESPACE ) available since MySQL 8.0.14. Percona XtraBackup also supports the binary log
encryption introduced in MySQL 8.0.14.
Two new options were added to xbstream. Use the --decompress option with xbstream to decompress
individual qpress files. With the --decompress-threads option, specify the number of threads to apply when
decompressing. Thanks to Rauli Ikonen for this contribution.
This release of Percona XtraBackup is a General Availability release ready for use in a production
environment.
• The deprecated innobackupex has been removed. Use the xtrabackup command to back up your
instances: $ xtrabackup --backup --target-dir=/data/backup
• When migrating from earlier database server versions, backup and restore and using Percona
XtraBackup 2.4 and then use mysql_upgrade from MySQL 8.0.x
• If using yum or apt repositories to install Percona Xtrabackup 8.0.5, ensure that you have enabled the
new tools repository. You can do this with the percona-release enable tools release command and
then install the percona-xtrabackup-80 package.
• PXB-1548: Percona XtraBackup enables updating the ib_buffer_pool file with the latest pages present
in the buffer pool using the --dump-innodb-buffer-pool option. Thanks to Marcelo Altmann for
contribution.
• PXB-1768: Added support for undo tablespaces created with the new MySQL 8.0.14 syntax.
• PXB-1781: Added support for binary log encryption introduced in MySQL 8.0.14.
• PXB-1797: For xbstream, two new options were added. The --decompress option enables xbstream to
decompress individual qpress files. The --decompress-threads option controls the number of threads to
apply when decompressing. Thanks to Rauli Ikonen for this contribution.
• Using --lock-ddl-per-table caused the server to scan all records of partitioned tables which could lead
to the “out of memory” error. Bugs fixed PXB-1691 and PXB-1698.
• When Percona XtraBackup was started run with the --slave-info , incorrect coordinates were written to
the xtrabackup_slave_info file.. Bug fixed PXB-1737
• Percona XtraBackup could crash at the prepare stage when making an incremental backup if the
variable innodb-rollback-segments was changed after starting the MySQL Server. Bug fixed PXB-1785.
• The full backup could fail when Percona Server for MySQL was started with the --innodb-encrypt-tables
parameter. Bug fixed PXB-1793.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
Percona XtraBackup enables MySQL backups without blocking user queries, making it ideal for companies
with large data sets and mission-critical applications that cannot tolerate long periods of downtime.
Offered free as an open source solution, it drives down backup costs while providing unique features for
MySQL backups.
This release of Percona Xtrabackup is a General Availability release ready for use in a production
environment.
• The deprecated innobackupex has been removed. Use the xtrabackup command to back up your
instances: $ xtrabackup --backup --target-dir=/data/backup
• When migrating from earlier database server versions, backup and restore and using Percona
XtraBackup 2.4 and then use mysql_upgrade from MySQL 8.0.x
• If using yum or apt repositories to install Percona XtraBackup 8.0.4, ensure that you have enabled the
new tools repository. You can do this with the percona-release enable tools release command and
then install the percona-xtrabackup-80 package.
All Percona software is open-source and free. We are grateful to the community for the invaluable
contributions to Percona XtraBackup. We would especially like to highlight the input of Alexey Kopytov who
has been actively offering improvements and submitting bug reports for Percona XtraBackup.
• Percona XtraBackup 8.0.4 is based on MySQL 8.0.13 and fully supports Percona Server for MySQL 8.0
series and MySQL 8.0 series.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
This is a Release Candidate quality release and it is not intended for production. If you want a high quality,
Generally Available release, use the current Stable version (the most recent stable version at the time of
writing is 2.4.12 in the 2.4 series).
Things to Note
• Due to the new MySQL redo log and data dictionary formats the Percona XtraBackup 8.0.x versions will
only be compatible with MySQL 8.0.x and the upcoming Percona Server for MySQL 8.0.x
• For experimental migrations from earlier database server versions, you will need to backup and restore
and using XtraBackup 2.4 and then use mysql_upgrade from MySQL 8.0.x
13.25.1 Improvements
• PXB-1678: Incremental backup prepare with the --apply-log-only option could roll back uncommitted
transactions
• PXB-1672: The MTS slave without GTID could be backed up when the --safe-slave-backup option was
applied.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
14.1 Error Message: Found tables with row versions due to INSTANT ADD/
DROP columns
MySQL 8.0.29 extended the support for ALTER TABLE … ALGORITHM=INSTANT to allow users to add a column in
any table position or drop any column. As part of this update, the redo log format has changed for all server
DML operations. This updated redo log format has a design flaw that can cause data corruption for tables
with INSTANT ADD/DROP COLUMNS.
The corruption happens when the crash recovery occurs. InnoDB applies redo logs at startup. Xtrabackup
copies the redo log during the backup and applies the log as part of the --prepare step to bring the backup
to a consistent state.
Percona fixed the corruption issue and several other issues with the INSTANT ADD/DROP column feature in
the upcoming Percona Server for MySQL 8.0.29.
• PS-8291,
• PS-8292
• PS-8303
Percona XtraBackup 8.0.32-25 allows instant columns in the backup for MySQL 8.0.32. This limitation remains
for backing up MySQL 8.0.31 and lower versions. This limitation does not exist for backing up Percona Server
for any version.
Percona XtraBackup 8.0.29 is able to take backups of Percona Server for MySQL 8.0.29 with tables that have
INSTANT ADD/DROP COLUMNS. However, the issues in MySQL 8.0.29 make this version unsafe to take backups.
It is impossible for XtraBackup to deal with the corrupted redo log generated by MySQL 8.0.29 and, for this
reason, XtraBackup 8.0.29 version will not take backups if it detects tables with INSTANT ADD/DROP COLUMNS
and generates an error listing the affected tables and providing instructions to convert them to regular
tables.
Please avoid ALTER ADD/DROP COLUMN without an explicit ALGORITHM=INPLACE. The default ALGORITHM is
INSTANT, so ALTER TABLE without the ALGORITHM keyword uses the newly introduced INSTANT algorithm.
If you already have such tables (information below on how to find such tables), users are advised to do
OPTIMIZE TABLE against these tables before taking backups.
To find tables with INSTANT ADD/DROP COLUMNS run the following command:
If this query shows an empty result set, you have no issues. Percona Xtrabackup takes the backup of your
MySQL 8.0.29 server. If the results are a list of tables, please run OPTIMIZE TABLE on the list before taking the
backup.
If Percona XtraBackup detects that MySQL 8.0.29 server has tables with instant add/drop columns, it aborts
with the following error message
14.1.2 Summary
The option, ALGORITHM=INSTANT is the default value in MySQL 8.0.29. If you do not specify an algorithm, all
ALTER TABLE ADD/DROP COLUMN statements use this algorithm.
Percona XtraBackup refuses to take backups from MySQL 8.0.29 tables that have used this algorithm.
Percona Server users do not suffer the same limitations as the corruption issues are fixed in the upcoming
release of Percona Server for MySQL 8.0.29
Contact us
For paid support and managed or consulting services , contact Percona Sales.
15. References
• --copy-back to copy data from a backup to the location that contained the original data; to move data
instead of copying use the alternate --move-back mode.
• --stats mode to scan the specified data files and print out index statistics.
When you intend to run xtrabackup in any of these modes, use the following syntax:
For all modes, the default options are read from the xtrabackup and mysqld configuration groups from the
following files in the given order:
1. /etc/my.cnf
2. /etc/mysql/my.cnf
3. /usr/etc/my.cnf
4. ~/.my.cnf .
As the first parameter to xtrabackup (in place of the --defaults-file , you may supply one of the following:
• --no-defaults to forbid reading options from any file but the login file.
• --defaults-file to read the default options from the given file.
• --defaults-extra-file to read the specified additional file after the global files have been read.
• --defaults-group-suffix to read the configuration groups with the given suffix. The effective group
name is constructed by concatenating the default configuration groups (xtrabackup and mysqld)
with the given suffix.
InnoDB options
There is a large group of InnoDB options that are normally read from the my.cnf configuration file, so that
xtrabackup boots up its embedded InnoDB in the same configuration as your current server. You normally
do not need to specify them explicitly. These options have the same behavior in InnoDB and XtraDB. See --
innodb-miscellaneous for more information.
15.1.2 Options
--apply-log-only()
This option causes only the redo stage to be performed when preparing a backup. It is very important for
incremental backups.
--backup()
--backup-lock-timeout()
--backup-lock-retry-count()
--backup-locks()
This option controls if backup locks should be used instead of FLUSH TABLES
WITH READ LOCK on the backup stage. The option has no effect when backup locks are not supported by the
server. This option is enabled by default, disable with --no-backup-locks .
--check-privileges()
This option checks if Percona XtraBackup has all required privileges. If a missing privilege is required for the
current operation, it will terminate and print out an error message. If a missing privilege is not required for
the current operation, but may be necessary for some other XtraBackup operation, the process is not
aborted and a warning is printed.
--close-files()
Do not keep files opened. When xtrabackup opens tablespace it normally doesn’t close its file handle in
order to handle the DDL operations correctly. However, if the number of tablespaces is really huge and can
not fit into any limit, there is an option to close file handles once they are no longer accessed. Percona
XtraBackup can produce inconsistent backups with this option enabled. Use at your own risk.
--compress()
This option tells xtrabackup to compress all output data, including the transaction log file and meta data
files, using either the quicklz , lz4 , or ZSTD compression algorithm. quicklz is chosen by default.
Note
Starting with Percona XtraBackup 8.0.31-24 using qpress/QuickLZ to compress backups is deprecated and may be
removed in future versions. We recommend using either LZ4 or Zstandard ( ZSTD ) compression algorithms. See
Compressed backups for more information.
When using --compress=quicklz or --compress , the resulting files have the qpress archive format. Every
\*.qp file produced by xtrabackup is essentially a one-file qpress archive and can be extracted and
uncompressed by the qpress file archiver.
--compress=lz4 produces \*.lz4 files. You can extract the contents of these files by using lz4 program.
--compress=zstd produces \*.zst files. You can extract the contents of these files by using the --decompress
option. You can specify ZSTD compression level with the --compress-zstd-level(=#) option.
--compress-chunk-size(=#)
Size of working buffer(s) for compression threads in bytes. The default value is 64K.
--compress-threads(=#)
This option specifies the number of worker threads used by xtrabackup for parallel data compression. This
option defaults to 1 . Parallel compression ( --compress-threads ) can be used together with parallel file
copying ( --parallel ). For example, --parallel=4 --compress --compress-threads=2 will create 4 I/O threads
that will read the data and pipe it to 2 compression threads.
--compress-zstd-level(=#)
This option is tech preview quality. Before using --compress-zstd-level(=#) in production, we recommend
that you test restoring production from physical backups in your environment, and also use the alternative
backup method for redundancy.
This option specifies ZSTD compression level. The default value is 1. Allowed range of values is from 1 to 19.
The option has been implemented in Percona XtraBackup 8.0.30-22.
--copy-back()
Copy all the files in a previously made backup from the backup directory to their original locations. This
option will not copy over existing files unless --force-non-empty-directories option is specified.
--core-file()
--databases(=#)
This option specifies a list of databases and tables that should be backed up. The option accepts the list of
the form "databasename1[.table_name1]
databasename2[.table_name2] . . ." .
--databases-exclude(=name)
Excluding databases based on name, Operates the same way as --databases , but matched names are
excluded from backup. Note that this option has a higher priority than --databases .
--databases-file(=#)
This option specifies the path to the file containing the list of databases and tables that should be backed
up. The file can contain the list elements of the form databasename1[.table_name1] , one element per line.
--datadir(=DIRECTORY)
The source directory for the backup. This should be the same as the datadir for your MySQL server, so it
should be read from my.cnf if that exists; otherwise you must specify it on the command line.
When combined with the --copy-back or --move-back option, --datadir refers to the destination directory.
Once connected to the server, in order to perform a backup you will need READ and EXECUTE permissions at
a filesystem level in the server’s datadir.
--debug-sleep-before-unlock(=#)
--debug-sync(=name)
The debug sync point. This option is only used by the xtrabackup test suite.
--decompress()
Decompresses all files in a backup previously made with the --compress option. The --parallel option will
allow multiple files to be decrypted simultaneously. In order to decompress, the qpress utility MUST be
installed and accessible within the path. Percona XtraBackup does not automatically remove the
compressed files. In order to clean up the backup directory users should use --remove-original option.
The --decompress option may be used with xbstream to decompress individual qpress files.
--decompress-threads(=#)
--decrypt(=ENCRYPTION-ALGORITHM)
Decrypts all files with the .xbcrypt extension in a backup previously made with --encrypt option. The --
parallel option will allow multiple files to be decrypted simultaneously. Percona XtraBackup doesn’t
automatically remove the encrypted files. In order to clean up the backup directory users should use --
remove-original option.
--defaults-extra-file(=[MY.CNF])
Read this file after the global files are read. Must be given as the first option on the command-line.
--defaults-file(=[MY.CNF])
Only read default options from the given file. Must be given as the first option on the command-line. Must be
a real file; it cannot be a symbolic link.
--defaults-group(=GROUP-NAME)
This option is to set the group which should be read from the configuration file. This is used by xtrabackup if
you use the --defaults-group option. It is needed for mysqld_multi deployments.
--defaults-group-suffix(=#)
--dump-innodb-buffer-pool()
This option controls whether or not a new dump of buffer pool content should be done.
With --dump-innodb-buffer-pool , xtrabackup makes a request to the server to start the buffer pool dump (it
takes some time to complete and is done in background) at the beginning of a backup provided the status
variable innodb_buffer_pool_dump_status reports that the dump has been completed.
If innodb_buffer_pool_dump_status reports that there is running dump of buffer pool, xtrabackup waits for the
dump to complete using the value of --dump-innodb-buffer-pool-timeout
The file ib_buffer_pool stores tablespace ID and page ID data used to warm up the buffer pool sooner.
--dump-innodb-buffer-pool-timeout()
--dump-innodb-buffer-pool-pct()
This option contains the percentage of the most recently used buffer pool pages to dump.
This option is effective if --dump-innodb-buffer-pool option is set to ON. If this option contains a value,
xtrabackup sets the MySQL system variable innodb_buffer_pool_dump_pct . As soon as the buffer pool dump
completes or it is stopped (see --dump-innodb-buffer-pool-timeout ), the value of the MySQL system variable
is restored.
--encrypt(=ENCRYPTION_ALGORITHM)
This option instructs xtrabackup to encrypt backup copies of InnoDB data files using the algorithm specified
in the ENCRYPTION_ALGORITHM. Currently supported algorithms are: AES128 , AES192 and AES256
--encrypt-key(=ENCRYPTION_KEY)
A proper length encryption key to use. It is not recommended to use this option where there is uncontrolled
access to the machine as the command line and thus the key can be viewed as part of the process info.
--encrypt-key-file(=ENCRYPTION_KEY_FILE)
The name of a file where the raw key of the appropriate length can be read from. The file must be a simple
binary (or text) file that contains exactly the key to be used.
It is passed directly to the xtrabackup child process. See the xtrabackup documentation for more details.
--encrypt-threads(=#)
This option specifies the number of worker threads that will be used for parallel encryption/decryption. See
the xtrabackup documentation for more details.
--encrypt-chunk-size(=#)
This option specifies the size of the internal working buffer for each encryption thread, measured in bytes. It
is passed directly to the xtrabackup child process.
To adjust the chunk size for encrypted files, use --read-buffer-size and --encrypt-chunk-size .
--estimate-memory(=#)
This option is in tech preview. Before using this option in production, we recommend that you test restoring
production from physical backups in your environment, and also use the alternative backup method for
redundancy.
Implemented in Percona XtraBackup 8.0.32-26, the option lets you enable or disable the Smart memory
estimation feature. The default value is OFF. Enable the feature by setting --estimate-memory=ON in the
backup phase and setting the --use-free-memory-pct option in the --prepare phase. If the
--estimate-memory setting is disabled, the --use-free-memory-pct setting is ignored.
--export()
Create files necessary for exporting tables. See Restoring Individual Tables.
--extra-lsndir(=DIRECTORY)
(for –backup): save an extra copy of the xtrabackup_checkpoints and xtrabackup_info files in this directory.
--force-non-empty-directories()
When specified, it makes --copy-back and --move-back option transfer files to non-empty directories. No
existing files will be overwritten. If files that need to be copied/moved from the backup directory already
exist in the destination directory, it will still fail with an error.
--ftwrl-wait-timeout(=SECONDS)
This option specifies time in seconds that xtrabackup should wait for queries that would block FLUSH TABLES
WITH READ LOCK before running it. If there are still such queries when the timeout expires, xtrabackup
terminates with an error. Default is 0 , in which case it does not wait for queries to complete and starts
FLUSH TABLES WITH READ LOCK immediately. Where supported xtrabackup will automatically use Backup
Locks as a lightweight alternative to FLUSH TABLES WITH READ LOCK to copy non-InnoDB data to avoid
blocking DML queries that modify InnoDB tables.
--ftwrl-wait-threshold(=SECONDS)
This option specifies the query run time threshold which is used by xtrabackup to detect long-running
queries with a non-zero value of --ftwrl-wait-timeout . FLUSH TABLES WITH READ LOCK is not started until such
long-running queries exist. This option has no effect if --ftwrl-wait-timeout is 0 . Default value is 60
seconds. Where supported xtrabackup will automatically use Backup Locks as a lightweight alternative to
FLUSH TABLES WITH READ LOCK to copy non-InnoDB data to avoid blocking DML queries that modify InnoDB
tables.
--ftwrl-wait-query-type(=all|update)
This option specifies which types of queries are allowed to complete before xtrabackup will issue the global
lock. Default is all .
--galera-info()
This option creates the xtrabackup_galera_info file which contains the local node state at the time of the
backup. Option should be used when performing the backup of Percona XtraDB Cluster. It has no effect
when backup locks are used to create the backup.
--generate-new-master-key()
--generate-transition-key()
xtrabackup needs to access the same keyring file or vault server during prepare and copy-back but it
should not depend on whether the server keys have been purged.
--generate-transition-key creates and adds to the keyring a transition key for xtrabackup to use if the
master key used for encryption is not found because it has been rotated and purged.
--get-server-public-key()
--help()
When run with this option or without any options xtrabackup displays information about how to run the
program on the command line along with all supported options and variables with default values where
appropriate.
--history(=NAME)
This option enables the tracking of backup history in the PERCONA_SCHEMA.xtrabackup_history table. An
optional history series name may be specified that will be placed with the history record for the current
backup being taken.
--host(=HOST)
This option accepts a string argument that specifies the host to use when connecting to the database
server with TCP/IP. It is passed to the mysql child process without alteration. See mysql --help for details.
--incremental()
This option tells xtrabackup to create an incremental backup. It is passed to the xtrabackup child process.
When this option is specified, either --incremental-lsn or --incremental-basedir can also be given. If neither
option is given, option --incremental-basedir is passed to xtrabackup by default, set to the first
timestamped backup directory in the backup base directory.
--incremental-basedir(=DIRECTORY)
When creating an incremental backup, this is the directory containing the full backup that is the base
dataset for the incremental backups.
--incremental-dir(=DIRECTORY)
When preparing an incremental backup, this is the directory where the incremental backup is combined
with the full backup to make a new full backup.
--incremental-force-scan()
When creating an incremental backup, force a full scan of the data pages in that instance.
--incremental-history-name(=name)
This option specifies the name of the backup series stored in the PERCONA_SCHEMA.xtrabackup_history history
record to base an incremental backup on. xtrabackup will search the history table looking for the most
recent (highest innodb_to_lsn ), successful backup in the series and take the to_lsn value to use as the
starting lsn for the incremental backup. This will be mutually exclusive with --incremental-history-uuid , --
incremental-basedir and --incremental-lsn . If no valid lsn can be found (no series by that name, no
successful backups by that name) xtrabackup will return with an error. It is used with the --incremental
option.
--incremental-history-uuid(=name)
This option specifies the UUID of the specific history record stored in the PERCONA_SCHEMA.xtrabackup_history
to base an incremental backup on. --incremental-history-name , --incremental-basedir and --incremental-
lsn . If no valid lsn can be found (no success record with that UUID) xtrabackup will return with an error. It is
used with the –incremental option.
--incremental-lsn(=LSN)
When creating an incremental backup, you can specify the log sequence number (LSN) instead of
specifying --incremental-basedir . For databases created in 5.1 and later, specify the LSN as a single 64-bit
integer. ATTENTION: If a wrong LSN value is specified (a user error which Percona XtraBackup is unable to
detect), the backup will be unusable. Be careful!
--innodb([=name])
--innodb-miscellaneous()
There is a large group of InnoDB options that are normally read from the my.cnf configuration file, so that
xtrabackup boots up its embedded InnoDB in the same configuration as your current server. You normally
do not need to specify these explicitly. These options have the same behavior in InnoDB and XtraDB:
--keyring-file-data(=FILENAME)
The path to the keyring file. Combine this option with --xtrabackup-plugin-dir .
--kill-long-queries-timeout(=SECONDS)
This option specifies the number of seconds xtrabackup waits between starting
FLUSH TABLES WITH READ LOCK and killing those queries that block it. Default is 0 seconds, which means
xtrabackup will not attempt to kill any queries. In order to use this option xtrabackup user should have the
PROCESS and SUPER privileges. Where supported, xtrabackup automatically uses Backup Locks as a
lightweight alternative to FLUSH TABLES WITH READ LOCK to copy non-InnoDB data to avoid blocking DML
queries that modify InnoDB tables.
--kill-long-query-type(=all|select)
This option specifies which types of queries should be killed to unblock the global lock. Default is “select”.
--lock-ddl()
Issue LOCK TABLES FOR BACKUP if it is supported by server (otherwise use LOCK INSTANCE FOR BACKUP ) at the
beginning of the backup to block all DDL operations.
Note
Prior to Percona XtraBackup 8.0.22-15.0, using a safe-slave-backup stops the SQL replica thread after the InnoDB
tables and before the non-InnoDB tables are backed up.
As of Percona XtraBackup 8.0.22-15.0, using a safe-slave-backup option stops the SQL replica thread before
copying the InnoDB files.
--lock-ddl-per-table()
Lock DDL for each table before xtrabackup starts to copy it and until the backup is completed.
NOTE: As of Percona XtraBackup 8.0.15, the –lock-ddl-per-table option is deprecated. Use the –lock-ddl
option instead.
--lock-ddl-timeout()
If LOCK TABLES FOR BACKUP or LOCK INSTANCE FOR BACKUP does not return within given timeout, abort the
backup.
--log()
--log-bin()
--log-bin-index(=name)
--log-copy-interval(=#)
This option specifies the time interval between checks done by the log copying thread in milliseconds
(default is 1 second).
--login-path()
--move-back()
Move all the files in a previously made backup from the backup directory to their original locations. As this
option removes backup files, it must be used with caution.
--no-backup-locks()
--no-defaults()
The default options are only read from the login file.
--no-lock()
Use this option to disable table lock with FLUSH TABLES WITH READ
LOCK . Use it only if ALL your tables are InnoDB and you DO NOT CARE about the binary log position of the
backup. This option shouldn’t be used if there are any DDL statements being executed or if any updates are
happening on non-InnoDB tables (this includes the system MyISAM tables in the mysql database), otherwise
it could lead to an inconsistent backup. Where supported xtrabackup will automatically use Backup Locks as
a lightweight alternative to FLUSH TABLES WITH READ LOCK to copy non-InnoDB data to avoid blocking DML
queries that modify InnoDB tables. If you are considering to use this because your backups are failing to
acquire the lock, this could be because of incoming replication events are preventing the lock from
succeeding. Please try using --safe-slave-backup to momentarily stop the replication replica thread, this
may help the backup to succeed and you do not need to use this option.
--no-server-version-check()
The default behavior runs a check that compares the source system version to the Percona XtraBackup
version. If the source system version is higher than the XtraBackup version, the backup is aborted with a
message.
Adding the option overrides this check, and the backup proceeds, but there may be issues with the backup.
See Server Version and Backup Version Comparison for more information.
--no-version-check()
This option disables the version check. If you do not pass this option, the automatic version check is enabled
implicitly when xtrabackup runs in the --backup mode. To disable the version check, you should pass
explicitly the --no-version-check option when invoking xtrabackup.
When the automatic version check is enabled, xtrabackup performs a version check against the server on
the backup stage after creating a server connection. xtrabackup sends the following information to the
server:
• Perl version
Each piece of information has a unique identifier. This is a MD5 hash value that Percona Toolkit uses to
obtain statistics about how it is used. This is a random UUID; no client information is either collected or
stored.
--open-files-limit(=#)
--parallel(=#)
This option specifies the number of threads to use to copy multiple data files concurrently when creating a
backup. The default value is 1 (i.e., no concurrent transfer). In Percona XtraBackup 2.3.10 and newer, this
option can be used with the --copy-back option to copy the user data files in parallel (redo logs and system
tablespaces are copied in the main thread).
--password(=PASSWORD)
This option specifies the password to use when connecting to the database. It accepts a string argument.
See mysql --help for details.
--plugin-load()
--port(=PORT)
This option accepts a string argument that specifies the port to use when connecting to the database
server with TCP/IP. It is passed to the mysql child process without alteration. See mysql --help for details.
--prepare()
Makes xtrabackup perform a recovery on a backup created with --backup , so that it is ready to use. See
preparing a backup.
--print-defaults()
Print the program argument list and exit. Must be given as the first option on the command-line.
--print-param()
Makes xtrabackup print out parameters that can be used for copying the data files back to their original
locations to restore them.
--read-buffer-size()
Set the read buffer size. The given value is scaled up to page size. The default size is 10MB. Use this option to
increase the xbcloud/xbstream chunk size from the default size. To adjust the chunk size for encrypted files,
use --read-buffer-size and --encrypt-chunk-size .
--rebuild-indexes()
Rebuilds indexes in a compact backup. This option only has effect when the --prepare and --rebuild-
threads options are provided.
--rebuild-threads(=#)
Uses the given number of threads to rebuild indexes in a compact backup. This option only has effect with
the --prepare and --rebuild-indexes options.
--register-redo-log-consumer()
The --register-redo-log-consumer parameter is disabled by default. When enabled, this parameter lets
Percona XtraBackup register as a redo log consumer at the start of the backup. The server does not remove
a redo log that Percona XtraBackup (the consumer) has not yet copied. The consumer reads the redo log
and manually advances the log sequence number (LSN). The server blocks the writes during the process.
Based on the redo log consumption, the server determines when it can purge the log.
--remove-original()
Implemented in Percona XtraBackup 2.4.6, this option when specified will remove .qp , .xbcrypt and
.qp.xbcrypt files after decryption and decompression.
--rocksdb-datadir()
--rocksdb-wal-dir()
--rocksdb-checkpoint-max-age()
The checkpoint cannot be older than this number of seconds when the backup completes.
--rocksdb-checkpoint-max-count()
Complete the backup even if the checkpoint age requirement has not been met after this number of
checkpoints.
--rollback-prepared-trx()
--rsync()
Uses the rsync utility to optimize local file transfers. When this option is specified, xtrabackup uses rsync to
copy all non-InnoDB files instead of spawning a separate cp for each file, which can be much faster for
servers with a large number of databases or tables. This option cannot be used together with --stream .
--safe-slave-backup()
When specified, xtrabackup will stop the replica SQL thread just before running FLUSH TABLES WITH READ LOCK
and wait to start backup until Slave_open_temp_tables in SHOW STATUS is zero. If there are no open temporary
tables, the backup will take place, otherwise the SQL thread will be started and stopped until there are no
open temporary tables. The backup will fail if Slave_open_temp_tables does not become zero after --safe-
slave-backup-timeout seconds. The replication SQL thread will be restarted when the backup finishes. This
option is implemented in order to deal with replicating temporary tables and isn’t necessary with Row-
Based-Replication.
Note
Prior to Percona XtraBackup 8.0.22-15.0, using a safe-slave-backup stops the SQL replica thread after the InnoDB
tables and before the non-InnoDB tables are backed up.
As of Percona XtraBackup 8.0.22-15.0, using a safe-slave-backup option stops the SQL replica thread before
copying the InnoDB files.
--safe-slave-backup-timeout(=SECONDS)
How many seconds --safe-slave-backup should wait for Slave_open_temp_tables to become zero. Defaults to
300 seconds.
--secure-auth()
Refuse client connecting to server if it uses old (pre-4.1.1) protocol. (Enabled by default; use –skip-secure-
auth to disable.)
--server-id(=#)
--server-public-key-path()
The file path to the server public RSA key in the PEM format.
--skip-tables-compatibility-check()
See --tables-compatibility-check .
--slave-info()
This option is useful when backing up a replication replica server. It prints the binary log position of the
source server. It also writes the binary log coordinates to the xtrabackup_slave_info file as a CHANGE MASTER
command. A new replica for this source can be set up by starting a replica server on this backup and
issuing a CHANGE MASTER command with the binary log position saved in the xtrabackup_slave_info file.
--socket()
This option accepts a string argument that specifies the socket to use when connecting to the local
database server with a UNIX domain socket. It is passed to the mysql child process without alteration. See
mysql --help for details.
--ssl()
--ssl-ca()
--ssl-capath()
--ssl-cert()
--ssl-cipher()
--ssl-crl()
Path of the file that contains certificate revocation lists. MySQL server documentation.
--ssl-crlpath()
--ssl-fips-mode()
SSL FIPS mode (applies only for OpenSSL); permitted values are: OFF, ON, STRICT.
--ssl-key()
--ssl-mode()
--ssl-verify-server-cert()
Verify server certificate Common Name value against host name used when connecting to server.
--stats()
Causes xtrabackup to scan the specified data files and print out index statistics.
--stream(=FORMAT)
Stream all backup files to the standard output in the specified format. Currently, this option only supports
the xbstream format.
--strict()
If this option is specified, xtrabackup fails with an error when invalid parameters are passed.
--tables(=name)
A regular expression against which the full tablename, in databasename.tablename format, is matched. If the
name matches, the table is backed up. See partial backups.
--tables-compatibility-check()
Enables the engine compatibility warning. The default value is ON. To disable the engine compatibility
warning use --skip-tables-compatibility-check .
--tables-exclude(=name)
Filtering by regexp for table names. Operates the same way as --tables , but matched names are excluded
from backup. Note that this option has a higher priority than --tables .
--tables-file(=name)
A file containing one table name per line, in databasename.tablename format. The backup will be limited to
the specified tables.
--target-dir(=DIRECTORY)
This option specifies the destination directory for the backup. If the directory does not exist, xtrabackup
creates it. If the directory does exist and is empty, xtrabackup will succeed. xtrabackup will not overwrite
existing files, however; it will fail with operating system error 17, file exists .
If this option is a relative path, it is interpreted as being relative to the current working directory from which
xtrabackup is executed.
In order to perform a backup, you need READ , WRITE , and EXECUTE permissions at a filesystem level for the
directory that you supply as the value of --target-dir .
--innodb-temp-tablespaces-dir(=DIRECTORY)
Directory where temp tablespace files live, this path can be absolute.
--throttle(=#)
This option limits the number of chunks copied per second. The chunk size is 10 MB.
--tls-ciphersuites()
--tls-version()
TLS version to use, permitted values are: TLSv1, TLSv1.1, TLSv1.2, TLSv1.3.
--tmpdir(=name)
Specify the directory that will be used to store temporary files during the backup
--transition-key(=name)
This option is used to enable processing the backup without accessing the keyring vault server. In this case,
xtrabackup derives the AES encryption key from the specified passphrase and uses it to encrypt tablespace
keys of tablespaces being backed up.
If --transition-key does not have any value, xtrabackup will ask for it. The same passphrase should be
specified for the --prepare command.
--use-free-memory-pct()
The --use-free-memory-pct is a tech preview option. Before using this option in production, we recommend
that you test restoring production from physical backups in your environment, and also use the alternative
backup method for redundancy.
Implemented in Percona XtraBackup 8.0.30-23, this option lets you configure the Smart memory estimation
feature. The option controlls the amount of free memory that can be used to --prepare a backup. The
default value is 0 (zero) which defines the option as disabled. For example, if you set --use-free-memory-
pct=50 , then 50% of the free memory is used to prepare a backup. The maximum allowed value is 100.
This option works, only if --estimate-memory option is enabled. If the --estimate-memory option is disabled, the
--use-free-memory-pct setting is ignored.
--use-memory()
This option affects how much memory is allocated and is similar to innodb_buffer_pool_size . This option is
only relevant in the --prepare phase or when analyzing statistics with --stats . The default value is 100MB.
The recommended value is between 1GB to 2GB. Multiple values are supported if you provide the unit (for
example, 1MB, 1M, 1GB, 1G).
--user(=USERNAME)
This option specifies the MySQL username used when connecting to the server, if that’s not the current user.
The option accepts a string argument. See mysql –help for details.
-v()
See --version
--version()
--xtrabackup-plugin-dir(=DIRNAME)
The absolute path to the directory that contains the keyring plugin.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
This utility has been modeled after The xbstream binary to perform encryption and decryption outside of
Percona XtraBackup. xbcrypt has following command line options:
-d(, --decrypt()
-i(, --input(=name)
Optional input file. If not specified, input will be read from standard input.
-o(, --output(=name)
Optional output file. If not specified, output will be written to standard output.
-a(, --encrypt-algo(=name)
Encryption algorithm.
-k(, --encrypt-key(=name)
Encryption key.
-f(, --encrypt-key-file(=name)
-s(, --encrypt-chunk-size(=#)
Size of working buffer for encryption in bytes. The default value is 64K.
--encrypt-threads(=#)
This option specifies the number of worker threads that will be used for parallel encryption/decryption.
-v(, --verbose()
Contact us
For paid support and managed or consulting services , contact Percona Sales.
• with the -x option it extracts files from the stream read from its standard input to the current directory
unless specified otherwise with the -c option. Support for parallel extraction with the --parallel option
has been implemented in Percona XtraBackup 2.4.7.
• with the -c option it streams files specified on the command line to its standard output.
• with the --decrypt=ALGO option specified xbstream will automatically decrypt encrypted files when
extracting input stream. Supported values for this option are: AES128 , AES192 , and AES256 . Either --
encrypt-key or --encrypt-key-file options must be specified to provide encryption key, but not both.
This option has been implemented in Percona XtraBackup 2.4.7.
• with the --encrypt-threads option you can specify the number of threads for parallel data encryption.
The default value is 1 . This option has been implemented in Percona XtraBackup 2.4.7.
• the --encrypt-key option is used to specify the encryption key that will be used. It can’t be used with --
encrypt-key-file option because they are mutually exclusive. This option has been implemented in
Percona XtraBackup 2.4.7.
• the --encrypt-key-file option is used to specify the file that contains the encryption key. It can’t be
used with --encrypt-key option. because they are mutually exclusive. This option has been
implemented in Percona XtraBackup 2.4.7.
The utility also tries to minimize its impact on the OS page cache by using the appropriate posix_fadvise()
calls when available.
When compression is enabled with xtrabackup all data is being compressed, including the transaction log
file and meta data files, using the specified compression algorithm. Percona XtraBackup supports the
following compression algorithms:
quicklz
Note
Starting with Percona XtraBackup 8.0.31-24 using qpress/QuickLZ to compress backups is deprecated and may be
removed in future versions. We recommend using either LZ4 or Zstandard ( ZSTD ) compression algorithms.
The resulting files have the qpress archive format. Every \*.qp file produced by xtrabackup is essentially a
one-file qpress archive and can be extracted and uncompressed by the qpress file archiver . This means
that there is no need to decompress entire backup to restore a single table as with tar.gz .
To decompress individual files, run xbstream with the --decompress option. You may control the number of
threads used for decompressing by passing the --decompress-threads option.
Also, files can be decompressed using the qpress tool. Qpress supports multi-threaded decompression.
lz4
To compress files using the lz4 compression algorithm, set --compress option to lz4 :
Zstandard (ZSTD)
The Zstandard (ZSTD) compression algorithm is a tech preview feature. Before using ZSTD in production, we
recommend that you test restoring production from physical backups in your environment, and also use the
alternative backup method for redundancy.
Percona XtraBackup 8.0.30-23 adds support for the Zstandard (ZSTD) compression algorithm. ZSTD is a fast
lossless compression algorithm that targets real-time compression scenarios and better compression
ratios.
To compress files using the ZSTD compression algorithm, use the --compress=zstd option. The resulting files
have the \*.zst format.
You can specify ZSTD compression level with the --compress-zstd-level(=#) option. The defaul value is 1 .
Contact us
For paid support and managed or consulting services , contact Percona Sales.
Percona XtraBackup 8.0 does not support making backups of databases created in versions prior to 8.0 of
MySQL, Percona Server for MySQL or Percona XtraDB Cluster. As the changes that MySQL 8.0 introduced in
data dictionaries, redo log and undo log are incompatible with previous versions, it is currently impossible
for Percona XtraBackup 8.0 to also support versions prior to 8.0.
Due to changes in MySQL 8.0.20 released by Oracle at the end of April 2020, Percona XtraBackup 8.0, up to
version 8.0.11, is not compatible with MySQL version 8.0.20 or higher, or Percona products that are based on it:
Percona Server for MySQL and Percona XtraDB Cluster.
For more information, see Percona XtraBackup 8.x and MySQL 8.0.20
innobackupex has been removed from Percona XtraBackup 8.0 in favor of xtrabackup.
15.4.3 Are you aware of any web-based backup management tools (commercial or not) built
around Percona XtraBackup?
Zmanda Recovery Manager is a commercial tool that uses Percona XtraBackup for Non-Blocking Backups:
“ZRM provides support for non-blocking backups of MySQL using Percona XtraBackup. ZRM with *Percona
XtraBackup provides resource utilization management by providing throttling based on the number of IO
operations per second. Percona XtraBackup based backups also allow for table level recovery even
though the backup was done at the database level (needs the recovery database server to be Percona
Server for MySQL with XtraDB).”*
In most of the cases this is due to not having installed the required libraries (and version) by xtrabackup.
Installing the GCC suite with the supporting libraries and recompiling xtrabackup solves the issue. See
Compiling and Installing from Source Code for instructions on the procedure.
15.4.5 How xtrabackup handles the ibdata/ib_log files on restore if they aren’t in mysql datadir?
In case the ibdata and ib_log files are located in different directories outside the datadir, you will have to
put them in their proper place after the logs have been applied.
15.4.6 Backup fails with Error 24: ‘Too many open files’
This usually happens when database being backed up contains large amount of files and Percona
XtraBackup can’t open all of them to create a successful backup. In order to avoid this error the operating
system should be configured appropriately so that Percona XtraBackup can open all its files. On Linux, this
can be done with the ulimit command for specific backup session or by editing the /etc/security/
limits.conf to change it globally (NOTE: the maximum possible value that can be set up is 1048576 which is
a hard-coded constant in the Linux kernel).
15.4.7 How to deal with skipping of redo logs for DDL operations?
To prevent creating corrupted backups when running DDL operations, Percona XtraBackup aborts if it
detects that redo logging is disabled. In this case, the following error is printed:
[FATAL] InnoDB: An optimized (without redo logging) DDL operation has been performed. All
modified pages may not have been flushed to the disk yet.
Percona XtraBackup will not be able to take a consistent backup. Retry the backup operation.
Note
Redo logging is disabled during a sorted index build. To avoid this error, Percona XtraBackup can use metadata
locks on tables while they are copied:
• To block all DDL operations, use the --lock-ddl option that issues LOCK TABLES FOR BACKUP .
• If LOCK TABLES FOR BACKUP is not supported, you can block DDL for each table before XtraBackup starts
to copy it and until the backup is completed using the --lock-ddl-per-table option.
Note
As of Percona XtraBackup 8.0.15, the –lock-ddl-per-table option is deprecated. Use the --lock-ddl option.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
15.5 Glossary
.CSM
Each table with the CSV Storage Engine has .CSM file which contains the metadata of it.
.CSV
Each table with the CSV Storage engine has .CSV file which contains the data of it (which is a standard
Comma Separated Value file).
.exp
Files with the .exp extension are created by Percona XtraBackup per each InnoDB tablespace when the --
export option is used on prepare. See restoring individual tables.
.frm
For each table, the server will create a file with the .frm extension containing the table definition (for all
storage engines).
A finalized version of the product which is made available to the general public. It is the final stage in the
software release cycle.
.ibd
On a multiple tablespace setup ([innodb_file_per_table] enabled), MySQL will store each newly created
table on a file with a .ibd extension.
.MRG
Each table using the MERGE storage engine, besides of a .frm file, will have .MRG file containing the names
of the MyISAM tables associated with it.
.MYD
Each MyISAM table has .MYD (MYData) file which contains the data on it.
.MYI
Each MyISAM table has .MYI (MYIndex) file which contains the table’s indexes.
.opt
MySQL stores options of a database (like charset) in a file with a .opt extension in the database directory.
.par
Each partitioned table has .par file which contains metadata about the partitions.
.TRG
The file contains the triggers associated with a table, for example, \mytable.TRG . With the .TRN file, they
represent all the trigger definitions.
.TRN
The file contains the names of triggers that are associated with a table, for example, \mytable.TRN . With the
.TRG file, they represent all the trigger definitions.
backup
compression
configuration file
crash
An unexpected shutdown which does not allow the normal server shutdown cleanup activities.
crash recovery
data dictionary
The metadata for the tables, indexes, and table columns stored in the InnoDB system tablespace.
datadir
The directory in which the database server stores its data files. Most Linux distribution use /var/lib/mysql by
default.
full backup
ibdata
The default prefix for tablespace files. For example, ibdata1 is a 10MB auto-extensible file that MySQL creates
for a shared tablespace by default.
incremental backup
InnoDB
Storage engine which provides ACID-compliant transactions and foreign key support, among others
improvements over MyISAM. It is the default engine for MySQL as of the 8.0 series.
innodb_buffer_pool_size
The size in bytes of the memory buffer to cache data and indexes of InnoDB’s tables. This aims to reduce
disk access to provide better performance.
[mysqld]
innodb_buffer_pool_size=8MB
innodb_data_home_dir
The directory (relative to datadir ) where the database server stores the files in a shared tablespace setup.
This option does not affect the location of innodb\_file\_per\_table . For example:
[mysqld]
innodb_data_home_dir = ./
innodb_data_file_path
[mysqld]
innodb_data_file_path=ibdata1:50M;ibdata2:50M:autoextend
innodb_file_per_table
By default, InnoDB creates tables and indexes in a file-per-tablespace. If the innodb_file_per_table variable
is disabled, you can enable the variable in your configuration file:
[mysqld]
innodb_file_per_table
or
start the server with --innodb_file_per_table .
innodb_log_group_home_dir
[mysqld]
innodb_log_group_home=/var/lib/mysql
logical backup
A backup which contains a set of SQL statements. The statements can be used to recreate the databases.
LSN
Each InnoDB page contains a log sequence number(LSN). The LSN is the system version number for the
database. Each page’s LSN shows how recently it was changed.
my.cnf
The database server’s main configuration file. Most Linux distributions place it as /etc/mysql/my.cnf or /etc/
my.cnf , but the location and name depends on the particular installation. Note that this method is not the
only way of configuring the server, some systems rely on the command options.
MyISAM
The MySQL default storage engine until version 5.5. It doesn’t fully support transactions but in some
scenarios may be faster than InnoDB. Each table is stored on disk in 3 files: .frm , .MYD , .MYI .
physical backup
This method restores the data into the state it was at any selected point of time.
prepared backup
restore
Copies the database backups taken using the backup command to the original location or a different
location. A restore returns data that has been either lost, corrupted, or stolen to the original condition at a
specific point in time.
Tech preview
A tech preview item can be a feature, a variable, or a value within a variable. The term designates that the
item is not yet ready for production use and is not included in support by SLA. A tech preview item is
included in a release so that users can provide feedback. The item is either updated and released as
general availability(GA) or removed if not useful. The item's functionality can change from tech preview to
GA.
xbcrypt
To support the encryption and the decryption of the backups, a new tool xbcrypt was introduced to Percona
XtraBackup. This utility has been modeled after the xbstream binary to perform encryption and decryption
outside Percona XtraBackup.
xbstream
To support simultaneous compression and streaming, Percona XtraBackup uses the xbstream format. For
more information see --stream
XtraDB
Percona XtraDB is an enhanced version of the InnoDB storage engine, designed to better scale on modern
hardware. Percona XtraDB includes features which are useful in a high performance environment. It is fully
backward-compatible, and is a drop-in replacement for the standard InnoDB storage engine. For more
information, see The Percona XtraDB Storage Engine.
ZSTD
The Zstandard (ZSTD) compression algorithm is a tech preview feature. Before using ZSTD in production, we
recommend that you test restoring production from physical backups in your environment, and also use the
alternative backup method for redundancy.
ZSTD is a fast lossless compression algorithm that targets real-time compression scenarios and better
compression ratios.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
backup-my.cnf This file contains information to start the mini instance of InnoDB during the
--prepare . This **not** a backup of the original my.cnf . The InnoDB
configuration is read from the file backup-my.cnf created by xtrabackup when
the backup was made. The --prepare uses InnoDB configuration from
backup-my.cnf by default, or from --defaults-file , if specified. The InnoDB's
configuration in this context means server variables that affect dataformat,
i.e. innodb_page_size option, innodb_log_block_size , etc. Location-related
variables, like innodb_log_group_home_dir or innodb_data_file_path are always
ignored by --prepare , so preparing a backup always works with data files
from the back directory, rather than any external ones.
xtrabackup_checkpoints The type of the backup (for example, full or incremental), its state (for
example, prepared) and the LSN range contained in it. This information is used
for incremental backups. Example of the xtrabackup_checkpoints after taking
a full backup:
backup_type = full-backuped
from lsn= 0
to_lsn = 15188961605
last_lsn = 15188961605
backup_type = incremental
from_lsn = 15188961605
to_lsn = 15189350111
last_lsn = 15189350111
xtrabackup_binlog_info The binary log file used by the server and its position at the moment of the
backup. A result of the following query:
xtrabackup_logfile Contains data needed for running the: --prepare . > The bigger this file is the
--prepare process > will take longer to finish.
<table_name>.delta.meta This file is going to be created when performing the incremental backup. > It
contains the per-table delta metadata: page size, size of compressed > page
(if the value is 0 it means the tablespace isn’t compressed) and > space id.
Example of this file:
page_size = 16384
zip_size = 0
space_id = 0
• Information related to the replication environment (if using the --slave-info option):
xtrabackup_slave_info
xtrabackup_galera_info
Contact us
For paid support and managed or consulting services , contact Percona Sales.
Percona owns a number of marks, including but not limited to Percona, XtraDB, Percona XtraDB, XtraBackup,
Percona XtraBackup, Percona Server, and Percona Live, plus the distinctive visual icons and logos
associated with these marks. Both the unregistered and registered marks of Percona are protected.
Use of any Percona trademark in the name, URL, or other identifying characteristic of any product, service,
website, or other use is not permitted without Percona’s written permission with the following three limited
exceptions.
First, you may use the appropriate Percona mark when making a nominative fair use reference to a bona
fide Percona product.
Second, when Percona has released a product under a version of the GNU General Public License (“GPL”),
you may use the appropriate Percona mark when distributing a verbatim copy of that product in
accordance with the terms and conditions of the GPL.
Third, you may use the appropriate Percona mark to refer to a distribution of GPL-released Percona software
that has been modified with minor changes for the sole purpose of allowing the software to operate on an
operating system or hardware platform for which Percona has not yet released the software, provided that
those third party changes do not affect the behavior, functionality, features, design or performance of the
software. Users who acquire this Percona-branded software receive substantially exact implementations of
the Percona software.
Percona reserves the right to revoke this authorization at any time in its sole discretion. For example, if
Percona believes that your modification is beyond the scope of the limited license granted in this Policy or
that your use of the Percona mark is detrimental to Percona, Percona will revoke this authorization. Upon
revocation, you must immediately cease using the applicable Percona mark. If you do not immediately
cease using the Percona mark upon revocation, Percona may take action to protect its rights and interests
in the Percona mark. Percona does not grant any license to use any Percona mark for any other modified
versions of Percona software; such use will require our prior written permission.
Neither trademark law nor any of the exceptions set forth in this Trademark Policy permit you to truncate,
modify or otherwise use any Percona mark as part of your own brand. For example, if XYZ creates a modified
version of the Percona Server, XYZ may not brand that modification as “XYZ Percona Server” or “Percona XYZ
Server”, even if that modification otherwise complies with the third exception noted above.
In all cases, you must comply with applicable law, the underlying license, and this Trademark Policy, as
amended from time to time. For instance, any mention of Percona trademarks should include the full
trademarked name, with proper spelling and capitalization, along with attribution of ownership to Percona
LLC and/or its affiliates. For example, the full proper name for XtraBackup is Percona XtraBackup. However, it
is acceptable to omit the word “Percona” for brevity on the second and subsequent uses, where such
omission does not cause confusion.
In the event of doubt as to any of the conditions or exceptions outlined in this Trademark Policy, please
contact trademarks@percona.com for assistance and we will do our very best to be helpful.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
Percona XtraBackup documentation is (C)2009-2023 Percona LLC and/or its affiliates and is distributed
under the Creative Commons Attribution 4.0 International License.
Contact us
For paid support and managed or consulting services , contact Percona Sales.
The purpose of this document is to articulate the information that is collected, as well as to provide
guidance on how to disable this functionality if desired.
15.9.1 Usage
Version Check was implemented in Percona Toolkit 2.1.4, and was enabled by default in version 2.2.1.
Currently, it is supported as a --[no]version-check option by a number of tools in Percona Toolkit, Percona
XtraBackup, and Percona Monitoring and Management (PMM).
When launched with Version Check enabled, the tool that supports this feature connects to a Percona’s
version check service via a secure HTTPS channel. It compares the locally installed version for possible
updates, and also checks versions of the following software:
• Operating System
• MySQL
• Perl
Then it checks for and warns about versions with known problems if they are identified as running in the
environment.
Each version check request is logged by the server. Stored information consists of the checked system
unique ID followed by the software name and version. The ID is generated either at installation or when the
version checking query is submitted for the first time.
Note
Prior to version 3.0.7 of Percona Toolkit, the system ID was calculated as an MD5 hash of a hostname, and starting
from Percona Toolkit 3.0.7 it is generated as an MD5 hash of a random number. Percona XtraBackup continues to
use hostname-based MD5 hash.
Expected output
{.text .no-copy}
85624f3fb5d2af8816178ea1493ed41a;DBD::mysql;4.044
c2b6d625ef3409164cbf8af4985c48d3;MySQL;MySQL Community Server (GPL) 5.7.22-log
85624f3fb5d2af8816178ea1493ed41a;OS;Manjaro Linux
85624f3fb5d2af8816178ea1493ed41a;Percona::Toolkit;3.0.11-dev
85624f3fb5d2af8816178ea1493ed41a;Perl;5.26.2
Although the version checking feature does not collect any personal information, you might prefer to
disable this feature, either one time or permanently. To disable it one time, use --no-version-check option
when invoking the tool from a Percona product which supports it. Here is a simple example which shows
running pt-diskstats tool from the Percona Toolkit with version checking turned off:
pt-diskstats --no-version-check
Disabling version checking permanently can be done by placing no-version-check option into the
configuration file of a Percona product (see correspondent documentation for exact file name and syntax).
For example, in case of Percona Toolkit this can be done in a global configuration file /etc/percona-toolkit/
percona-toolkit.conf :
In case of Percona XtraBackup this can be done in its configuration file in a similar way:
[xtrabackup]
no-version-check
We believe having this functionality enabled improves security and performance of environments running
Percona Software and it is good choice for majority of the users.
Why not rely on Operating System’s built in functionality for software updates?
In many environments the Operating Systems repositories may not carry the latest version of software and
newer versions of software often installed manually, so not being covered by operating system wide check
for updates.
Why do you send more information than just the version of software being run as a part of version check service?
Compatibility problems can be caused by versions of various components in the environment, for example
problematic versions of Perl, DBD or MySQL could cause operational problems with Percona Toolkit.
Contact us
For paid support and managed or consulting services , contact Percona Sales.