xtrabackup/innobackupex 最优方式


xtrabackup采用stream备份方式,传送到远端机器;

原先的流程是:磁盘读取-->打包-->传输-->落盘

采用“边压缩边传输”的方式(最重要还是要选好工具)

流程分为: 磁盘读取,tar打包,压缩 ,传输,[解压缩 ,拆包 可选],落盘

最理想的情况下是,几个关键步骤都能达到速度的顶峰,提升速度最慢的那个(解压缩过程),减少“木桶效应”的影响


xtrabackup远程备份可改为:

/usr/bin/innobackupex --slave-info --safe-slave-backup --user=mysqlbackup --password=xxx  --defaults-file=xxx --port=xxx --socket=xxx  --stream=tar /tmp/ 2>>/tmp/mysqlbackup.log |lz4 -B4|  ssh -c aes192-cbc -o "MACs umac-64@openssh.com"  mysqlbackup@xxx  "lz4 -d | tar -xiC /data/xxxxxxx

lz4压缩使用-B4(64KB块大小),解压使用-B7(4MB块大小),是测试最优值,  -B#     block size [4-7](default : 7)  B4= 64KB ; B5= 256KB ; B6= 1MB ; B7= 4MB
如果无需解压的传输,则可以考虑使用pigz/pbiz2


用流备份,默认的临时目录都是系统的/tmp目录,需要保证该目录有足够的空间,或指定--tmpdir选项

权限

create user 'dbabackup_umysql'@'localhost' IDENTIFIED BY 'dbabackup_pmysql';
GRANT SELECT, RELOAD, LOCK TABLES, PROCESS, SUPER, CREATE TABLESPACE, EVENT, SHOW VIEW ON *.* TO 'dbabackup_u'@'dbabackup_umysql';
# 如果使用 --history=PERCONA_SCHEMA.xtrabackup_history , 可以指定database?
GRANT CREATE,INSERT ON PERCONA_SCHEMA.xtrabackup_history.* TO 'dbabackup_u'@'dbabackup_umysql';


参数

--no-lock  可以在备份non-innodb和frm文件时不锁表,但是master和slave的pos信息就无法记录,因为write_binlog_info和write_slave_info函数只在mysql_lockall函数中调用,而mysql_lockal调用了flush tables with read lock (不适合生产环境,但如果都是innodb表并且当时没有DDL,则可以

--kill-long-queries-timeout=SECONDS
This option specifies the number of seconds innobackupex waits between starting FLUSH TABLES WITH READ LOCK and killing those queries that block it. Default is 0 seconds, which means innobackupex will not attempt to kill any queries. In order to use this option xtrabackup user should have PROCESS and SUPER privileges.
 FLUSH TABLES WITH READ LOCK 等待 N秒后,将kill阻塞的query。默认为0,表示不kill
--kill-long-query-type=all|select
This option specifies which types of queries should be killed to unblock the global lock. Default is “all”.
kill的query类型,默认type

--safe-slave-backup  
When specified, innobackupex will stop the slave SQL thread just before running FLUSH TABLES WITH READ LOCK and wait to start backup until Slave_open_temp_tables in SHOW STATUS is zero. If there are no open temporary tables, the backup will take place, otherwise the SQL thread will be started and stopped until there are no open temporary tables. The backup will fail if Slave_open_temp_tables does not become zero after innobackupex --safe-slave-backup-timeout seconds. The slave SQL thread will be restarted when the backup finishes.
FLUSH TABLES WITH READ LOCK前关闭 slave_sql_thread , 等待slave_open_temp_tables变为0(每隔3s检查一次),如果超过 --safe-slave-backup-timeout(默认300s) 仍然不为0,则备份失败。备份结束后重启 slave_sql_thread 
session中仍存在临时表时备份是否会对备份集可用性产生影响(还需要测试)
--safe-slave-backup-timeout=SECONDS
How many seconds innobackupex --safe-slave-backup should wait for Slave_open_temp_tables to become zero. Defaults to 300 seconds.

# 性能
--parallel=NUMBER-OF-THREADS
This option accepts an integer argument that specifies the number of threads the xtrabackup child process should use to back up files concurrently. Note that this option works on file level, that is, if you have several .ibd files, they will be copied in parallel. If your tables are stored together in a single tablespace file, it will have no effect. This option will allow multiple files to be decrypted and/or decompressed simultaneously. In order to decompress, the qpress utility MUST be installed and accessable within the path. This process will remove the original compressed/encrypted files and leave the results in the same location. It is passed directly to xtrabackup’s xtrabackup --parallel option. See the xtrabackup documentation for details
并发线程数

--throttle=IOS
This option accepts an integer argument that specifies the number of I/O operations (i.e., pairs of read+write) per second. It is passed directly to xtrabackup’s xtrabackup --throttle option. NOTE: This option works only during the backup phase, ie. it will not work with innobackupex --apply-log and innobackupex --copy-back options.
 this option limits the number of I/O operations per second in 1 MB units.
限制每秒磁盘读写量n(Mb),而不是读写次数
备份时IO读写限制,备份业务产生的IO是影响
--rsync
Uses the rsync utility to optimize local file transfers. When this option is specified, innobackupex uses rsync to copy all non-InnoDB files instead of spawning a separate cp for each file, which can be much faster for servers with a large number of databases or tables. This option cannot be used together with --stream.
在拷贝非innodb表时使用rsync代替cp,提高速度,但 不能与 --stream 一起使用

# 远程传送脚本
--remote-host=REMOTEUSER@REMOTEHOST /path/IN/REMOTE/HOST/to/backup/
将备份结果实时传送到 remote-host 的 /path/IN/REMOTE/HOST/to/backup/ 目录下
--scpopt = SCP-OPTIONS
This option accepts a string argument that specifies the command line options to pass to scp when the option --remost-host is specified. If the option is not specified, the default options are -Cp -c arcfour.
配合 --remost-host使用,scp的选项,默认 '-Cp -c arcfour'

Partial backup

Do Partial Backups

# --databases  空格分隔,可以指定多个库
innobackupex --databases="mydatabase.mytable mysql" /path/to/backup
# --include
    innobackupex --include='^mydatabase[.]mytable' /path/to/backup
# --tables-file, /tmp/tables.txt 每个表一行,格式为databasename.tablename, 支持通配符?应该支持
innobackupex --tables-file=/tmp/tables.txt /path/to/backup

# --tables-file 比较稳定,其他两个多个库备份时比较麻烦
备份时先将表导出到文件
fDoTables="$dirBak/doTables.cnf"
OPTMY=" --user=$bakuser --password=$bakpwd --socket=$socket"
qsql="SELECT CONCAT_WS('.',TABLE_SCHEMA, TABLE_NAME) AS ds FROM information_schema.TABLES WHERE TABLE_SCHEMA IN ('db1','db2')"
/usr/local/mysql/bin/mysql $OPTMY --silent --skip-column-names -e "$qsql" > $fDoTables
echo "$fDoTables"

Preparing Partial Backups

# 必须加--export
# 会产生warn信息,没有备份的表会被自动移除
innobackupex --apply-log --export /path/to/partial/backup

# 如果没有备份系统库mysql,则需要使用mysql_install_db生成

restore Partial backup

https://www.percona.com/doc/percona-xtrabackup/2.3/innobackupex/restoring_individual_tables_ibk.html


BUG

FLUSH TABLES WITH READ LOCK 超时退出
    可能原因:
        备库上长查询外,包含insert和update的长transaction也有可能引发这个问题

参考:

https://www.percona.com/doc/percona-xtrabackup/2.3/index.html#
压缩工具比较: http://blog.itpub.net/26250550/viewspace-1158863/ 
Innobackupex超时退出 和实现原理 : http://blog.chinaunix.net/uid-28212952-id-3420472.html


请使用浏览器的分享功能分享到微信等