{"id":95102,"date":"2022-11-22T08:54:08","date_gmt":"2022-11-22T08:54:08","guid":{"rendered":"https:\/\/www.red-gate.com\/simple-talk\/?p=95102"},"modified":"2022-11-22T08:54:08","modified_gmt":"2022-11-22T08:54:08","slug":"backing-up-mysql-part-4-mydumper","status":"publish","type":"post","link":"https:\/\/www.red-gate.com\/simple-talk\/blogs\/backing-up-mysql-part-4-mydumper\/","title":{"rendered":"Backing up MySQL Part 4: mydumper"},"content":{"rendered":"<p><span style=\"font-weight: 400\">There are many tools we can use to back up our MySQL databases. Some are well-known and used by the best technology companies out there (<\/span><a href=\"https:\/\/www.red-gate.com\/simple-talk\/databases\/mysql\/backing-up-mysql-part-1-mysqldump\/\"><code><span style=\"font-weight: 400\">mysqldump<\/span><\/code><span style=\"font-weight: 400\"> comes to mind<\/span><\/a><span style=\"font-weight: 400\">), and some are a little less famous, but still have their place in the MySQL world. Enter <\/span><code><a href=\"https:\/\/github.com\/mydumper\/mydumper\"><span style=\"font-weight: 400\">mydumper<\/span><\/a><\/code><span style=\"font-weight: 400\"> \u2013 the tool is built by the engineering team over at Percona and it\u2018s supposedly created to address performance issues caused by <\/span><code><span style=\"font-weight: 400\">mysqldump<\/span><\/code><span style=\"font-weight: 400\">.<\/span><\/p>\n<h2><span style=\"font-weight: 400\">Prerequisites and Installation<\/span><\/h2>\n<p><code><span style=\"font-weight: 400\">mydumper<\/span><\/code><span style=\"font-weight: 400\"> is a tool that lets people take logical backups of their MySQL databases. The tool is built according to the best MySQL practices, it\u2018s being actively maintained, and works on all flavors of MySQL, including Percona Server and MariaDB. Note that the tool is not built-in to MySQL \u2013 to use it, one needs to install the <\/span><i><span style=\"font-weight: 400\">development versions of MySQL<\/span><\/i><span style=\"font-weight: 400\"> or any of its flavors, install a couple of development tools, and install the development versions of <\/span><a href=\"https:\/\/docs.gtk.org\/glib\/\"><span style=\"font-weight: 400\">GLib<\/span><\/a><span style=\"font-weight: 400\">, <\/span><a href=\"https:\/\/github.com\/madler\/zlib\"><span style=\"font-weight: 400\">ZLib<\/span><\/a><span style=\"font-weight: 400\">, <\/span><a href=\"https:\/\/www.pcre.org\/\"><span style=\"font-weight: 400\">PCRE<\/span><\/a><span style=\"font-weight: 400\">, and <\/span><a href=\"https:\/\/github.com\/facebook\/zstd\"><span style=\"font-weight: 400\">ZSTD<\/span><\/a><span style=\"font-weight: 400\"> \u2013 once all of these libraries are installed, we should follow the instructions for installation that are available <\/span><a href=\"https:\/\/github.com\/mydumper\/mydumper\"><span style=\"font-weight: 400\">here<\/span><\/a><span style=\"font-weight: 400\">.<\/span><\/p>\n<p><span style=\"font-weight: 400\">All of these libraries accomplish specific goals \u2013 GLib is a bundle of libraries, ZLib is used to compress data, PCRE stands for \u201ePerl-Compatible Regular Expressions\u201c, and ZSTD \u2013 otherwise known as ZStandard \u2013 is a compression algorithm developed by Facebook.<\/span><\/p>\n<p><span style=\"font-weight: 400\">The installation of <\/span><code><span style=\"font-weight: 400\">mydumper<\/span><\/code><span style=\"font-weight: 400\"> will provide us with:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">The ability to export and import data using multiple threads in parallel, sort of like <\/span><code><span style=\"font-weight: 400\">mysqlpump<\/span><\/code><span style=\"font-weight: 400\"> does.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">The ability to use regex expressions when exporting data.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Easily manageable output \u2013 all tables will be backed up into separate files exclusive to them, data and metadata will be separated, etc.<\/span><\/li>\n<\/ul>\n<h2><span style=\"font-weight: 400\">Using mydumper<\/span><\/h2>\n<p><span style=\"font-weight: 400\">Once <\/span><code><span style=\"font-weight: 400\">mydumper<\/span><\/code><span style=\"font-weight: 400\"> is installed, we can start playing with the tool. Even though there are multiple advanced features provided by the tool, the syntax isn\u2018t rocket science and looks like the following:<\/span><\/p>\n<p><code><span style=\"font-weight: 400\">mydumper [options]<\/span><\/code><\/p>\n<p><span style=\"font-weight: 400\">That\u2018s it, really. Well, users will still need to specify the host (<\/span><code><span style=\"font-weight: 400\">--host<\/span><\/code><span style=\"font-weight: 400\">), user (<\/span><code><span style=\"font-weight: 400\">--user<\/span><\/code><span style=\"font-weight: 400\">), and password (<\/span><code><span style=\"font-weight: 400\">--password<\/span><\/code><span style=\"font-weight: 400\">) to work with, but other than that, only the options remain. The output that is received will, of course, vary and depend directly on the options that are specified, so let\u2018s look into them as well:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Users can specify the <\/span><code><span style=\"font-weight: 400\">--database<\/span><\/code><span style=\"font-weight: 400\"> option to provide <\/span><code><span style=\"font-weight: 400\">mydumper<\/span><\/code><span style=\"font-weight: 400\"> with the database to work with (dump data from.) Alternatively, the <\/span><code><span style=\"font-weight: 400\">--B<\/span><\/code><span style=\"font-weight: 400\"> option can also be specified.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">The <\/span><code><span style=\"font-weight: 400\">--tables<\/span><\/code><span style=\"font-weight: 400\"> option, or its shortened version <\/span><code><span style=\"font-weight: 400\">--T<\/span><\/code><span style=\"font-weight: 400\">, will let users specify a comma-delimited list of tables to take a copy of.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">The databases and tables that are being dumped need to have an output directory that can be specified by appending <\/span><code><span style=\"font-weight: 400\">--outputdir<\/span><\/code><span style=\"font-weight: 400\"> or <\/span><code><span style=\"font-weight: 400\">--o<\/span><\/code><span style=\"font-weight: 400\"> (the directory must be writable.)<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Users can also split <\/span><code><span style=\"font-weight: 400\">INSERT<\/span><\/code><span style=\"font-weight: 400\"> statements into smaller size by using <\/span><code><span style=\"font-weight: 400\">--statement-size<\/span><\/code><span style=\"font-weight: 400\"> (or <\/span><span style=\"font-weight: 400\"><code>--s<\/code>.<\/span><span style=\"font-weight: 400\">) Note that the size of <\/span><code><span style=\"font-weight: 400\">INSERT<\/span><\/code><span style=\"font-weight: 400\"> statements are to be specified in bytes (default size \u2013 1,000,000 bytes.)<\/span><\/li>\n<li style=\"font-weight: 400\"><code><span style=\"font-weight: 400\">mydumper<\/span><\/code><span style=\"font-weight: 400\"> allows its users to specify which storage engines to ignore (i.e. <\/span><code><span style=\"font-weight: 400\">mydumper<\/span><\/code><span style=\"font-weight: 400\"> will not back up tables built on this specific storage engine) by using <\/span><code><span style=\"font-weight: 400\">--ignore-engines<\/span><\/code><span style=\"font-weight: 400\"> or <\/span><code><span style=\"font-weight: 400\">--i<\/span><\/code><span style=\"font-weight: 400\">.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">The <\/span><code><span style=\"font-weight: 400\">--regex<\/span><\/code><span style=\"font-weight: 400\"> or <\/span><code><span style=\"font-weight: 400\">--r<\/span><\/code><span style=\"font-weight: 400\"> option will let users specify databases and tables using regex in the format of \u201e<\/span><span style=\"font-weight: 400\">database.table<\/span><span style=\"font-weight: 400\">.\u201c Something like this should do:<\/span><span style=\"font-weight: 400\"><br \/>\n<\/span><code><span style=\"font-weight: 400\">mydumper --regex \u201e(^demo_db\\.demo_database$)\u201c<\/span><\/code><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">To specify the number of threads <\/span><code><span style=\"font-weight: 400\">mydumper<\/span><\/code><span style=\"font-weight: 400\"> is able to use (the default number of allocated threads is 4), consider the <\/span><code><span style=\"font-weight: 400\">--threads<\/span><\/code><span style=\"font-weight: 400\"> or <\/span><code><span style=\"font-weight: 400\">--t<\/span><\/code><span style=\"font-weight: 400\"> option.<\/span><\/li>\n<li style=\"font-weight: 400\"><code><span style=\"font-weight: 400\">mydumper<\/span><\/code><span style=\"font-weight: 400\"> is also able to automatically send long-running queries to the timeout realm or kill long-running queries: that can be done by either specifying the <\/span><code><span style=\"font-weight: 400\">--long-query-guard<\/span><\/code><span style=\"font-weight: 400\"> or <\/span><code><span style=\"font-weight: 400\">--kill-long-queries<\/span><\/code><span style=\"font-weight: 400\"> parameters (or their shorter counterparts which are <\/span><code><span style=\"font-weight: 400\">--l<\/span><\/code><span style=\"font-weight: 400\"> and <\/span><code><span style=\"font-weight: 400\">--k<\/span><\/code><span style=\"font-weight: 400\"> respectively.)<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400\">The options specified above should tell you a little about just how powerful <\/span><code><span style=\"font-weight: 400\">mydumper<\/span><\/code><span style=\"font-weight: 400\"> can be when used properly \u2013 however, do note that <\/span><a href=\"https:\/\/github.com\/mydumper\/mydumper\/blob\/master\/docs\/examples.rst\"><span style=\"font-weight: 400\">other options can be used as well<\/span><\/a><span style=\"font-weight: 400\"> \u2013 <\/span><code><span style=\"font-weight: 400\">mydumper<\/span><\/code><span style=\"font-weight: 400\"> also comes with a brother called <\/span><code><span style=\"font-weight: 400\">myloader<\/span><\/code><span style=\"font-weight: 400\"> which is essentially a backup tool that restores data generated by <\/span><code><span style=\"font-weight: 400\">mydumper<\/span><\/code><span style=\"font-weight: 400\"> from a directory specified by the user. To use <\/span><code><span style=\"font-weight: 400\">myloader<\/span><\/code><span style=\"font-weight: 400\">, invoke it like so:<\/span><\/p>\n<p><code><span style=\"font-weight: 400\">myloader --directory=[directory] [--overwrite_tables] --user=your_user<\/span><\/code><\/p>\n<p><span style=\"font-weight: 400\">The <\/span><code><span style=\"font-weight: 400\">--directory<\/span><\/code><span style=\"font-weight: 400\"> option takes the directory inside of which the backup created by <\/span><code><span style=\"font-weight: 400\">mydumper<\/span><\/code><span style=\"font-weight: 400\"> is being stored, and the <\/span><code><span style=\"font-weight: 400\">overwrite-tables<\/span><\/code><span style=\"font-weight: 400\"> option can also be invoked to overwrite all existing tables.<\/span><\/p>\n<p><span style=\"font-weight: 400\">Make sure to read up on the docs surrounding the two tools <\/span><a href=\"https:\/\/github.com\/mydumper\/mydumper\/blob\/master\/docs\/examples.rst\"><span style=\"font-weight: 400\">here<\/span><\/a><span style=\"font-weight: 400\"> and follow <\/span><a href=\"https:\/\/www.percona.com\/blog\/\"><span style=\"font-weight: 400\">Percona\u2019s blog<\/span><\/a><span style=\"font-weight: 400\"> to learn more about the tools they develop \u2013 <a href=\"https:\/\/www.red-gate.com\/simple-talk\/databases\/\">explore the Databases part of SimpleTalk<\/a> to learn more about the functionality of all kinds of databases (our blog is not limited to MySQL!) and how they impact the behavior of applications, and we\u2019ll see you in the next one.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>In this blog, we walk you through one of the most prominent MySQL logical backup solutions &#8211; a CLI-based tool called mydumper.&hellip;<\/p>\n","protected":false},"author":339547,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[2],"tags":[5564,4168,5854,6011],"coauthors":[146040],"class_list":["post-95102","post","type-post","status-publish","format-standard","hentry","category-blogs","tag-backups","tag-database","tag-mysql","tag-software-tools"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/95102","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/users\/339547"}],"replies":[{"embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/comments?post=95102"}],"version-history":[{"count":4,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/95102\/revisions"}],"predecessor-version":[{"id":95106,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/95102\/revisions\/95106"}],"wp:attachment":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/media?parent=95102"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/categories?post=95102"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/tags?post=95102"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/coauthors?post=95102"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}