{"id":107551,"date":"2025-08-18T11:00:00","date_gmt":"2025-08-18T11:00:00","guid":{"rendered":"https:\/\/www.red-gate.com\/simple-talk\/?p=107551"},"modified":"2025-08-07T12:37:02","modified_gmt":"2025-08-07T12:37:02","slug":"mysql-shell-basic-configuration-management-part-6-backups-and-recovery-procedures","status":"publish","type":"post","link":"https:\/\/www.red-gate.com\/simple-talk\/databases\/mysql\/mysql-shell-basic-configuration-management-part-6-backups-and-recovery-procedures\/","title":{"rendered":"MySQL Shell Basic Configuration Management (Part 6 &#8211; Backups and Recovery Procedures)"},"content":{"rendered":"\n<p><em>In this final part of the <a href=\"https:\/\/www.red-gate.com\/simple-talk\/collections\/chisom-kanus-series-mysql-shell-and-basic-configuration-management\/\" target=\"_blank\" rel=\"noreferrer noopener\">MySQL Shell Configuration Management series<\/a>, we explore how to automate performance tuning, backups and recovery using <a href=\"https:\/\/dev.mysql.com\/doc\/mysql-shell\/8.0\/en\/\" target=\"_blank\" rel=\"noreferrer noopener\">MySQL Shell<\/a>. This guide introduces a complete system that captures not just data, but also configuration and performance context \u2014 enabling intelligent restores and optimized deployments across diverse environments.<\/em><\/p>\n\n\n\n<p>We&#8217;re going to focus on a commonly overlooked aspect of <a href=\"https:\/\/www.red-gate.com\/blog\/database-devops\/what-is-database-lifecycle-management\" target=\"_blank\" rel=\"noreferrer noopener\">database lifecycle management (DLM)<\/a>: creating versioned, performance-aware database copies for development and testing.<\/p>\n\n\n\n<p>These copies aren\u2019t just backups \u2014 they capture configuration and performance context, so that developers and testers can spin up environments with confidence, even on unfamiliar or scaled-down hardware. To illustrate the point, we\u2019ll share scripts that can provide a foundation for a system that captures not just your data but the entire context of your database environment, including configuration snapshots and performance baselines.<\/p>\n\n\n\n<p>It\u2019s the Recovery process that best displays the advantage of&nbsp;a DLM approach that preserves the whole database context. Instead of the na\u00efve optimism of restoring a <a href=\"https:\/\/www.red-gate.com\/simple-talk\/databases\/sql-server\/database-administration-sql-server\/developing-a-backup-plan\/\" target=\"_blank\" rel=\"noreferrer noopener\">backup<\/a> with its original configuration, we\u2019ll show how to build recovery procedures that adapt intelligently to the target environment. <\/p>\n\n\n\n<p>Imagine, for example, restoring a database from a server with 32GB of RAM to one with 64GB. Our system will automatically adjust buffer pool sizes, connection limits, and other parameters to match the capabilities of the new environment.<\/p>\n\n\n\n<p>Finally, we&#8217;ll bring everything together into an automation script that runs continuously, monitoring performance, making adjustments, generating reports, and maintaining backups. By the end of this article, you&#8217;ll have a complete automation system that can manage your <a href=\"https:\/\/www.mysql.com\/\" target=\"_blank\" rel=\"noreferrer noopener\">MySQL<\/a> servers from initial configuration through ongoing optimization, backup, and recovery. <\/p>\n\n\n\n<p>As well as providing resilience and accurate recovery, this will make it far easier to create reliable new instances of the database for development work in a variety of server environments. It also brings together in a single system everything we&#8217;ve learned throughout this series on <a href=\"https:\/\/www.red-gate.com\/simple-talk\/collections\/chisom-kanus-series-mysql-shell-and-basic-configuration-management\/\" target=\"_blank\" rel=\"noreferrer noopener\">MySQL Configuration Management<\/a>.<\/p>\n\n\n\n<p><em>Every script used in this article can be found in this <a href=\"https:\/\/github.com\/chisommmy\/mysql-shell-performance\" target=\"_blank\" rel=\"noreferrer noopener\">GitHub repo<\/a>.<\/em><\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-automating-backups-with-configuration-tracking-in-mysql-shell\">Automating Backups with Configuration Tracking in MySQL Shell<\/h2>\n\n\n\n<p>Backups are an important part of database management. Let&#8217;s look at a script that not only backs up your database but also records your configuration and performance data alongside the backup. When we restore or copy the database, we will have more information with which to calculate the configuration parameters, but also to test the result to make sure we have an installation which performs as we expect<\/p>\n\n\n\n<p>The function begins by creating a timestamp-based directory name for the backup, announcing the start of the backup process, creating the backup directory, and using MySQL Shell&#8217;s built-in `<strong>util.dumpInstance()`<\/strong> function to create a consistent backup of all databases:<\/p>\n\n\n\n<div class=\"wp-block-urvanov-syntax-highlighter-code-block\"><pre class=\"lang:tsql decode:true \">function backupWithConfig() {\n    var session = shell.getSession();\n    var timestamp = new Date().toISOString().replace(\/[:.]\/g, '-');\n    var backupDir = '\/tmp\/mysql_backup_' + timestamp;\n    print(\"Starting backup with configuration tracking...\");\n    print(\"Backup directory: \" + backupDir);\n    \n    \/\/ Create backup directory\n    shell.run('mkdir -p ' + backupDir);\n    \/\/ Backup the database using util.dumpInstance\n    util.dumpInstance(backupDir + '\/data', {\n        consistent: true,\n        compression: \"gzip\"\n    });<\/pre><\/div>\n\n\n\n<p>The `consistent: true` option ensures that the backup is transactionally consistent, which is essential for <a href=\"https:\/\/dev.mysql.com\/doc\/refman\/8.4\/en\/innodb-introduction.html\" target=\"_blank\" rel=\"noreferrer noopener\">InnoDB<\/a> tables.<\/p>\n\n\n\n<p>Next, the function creates a file to store the current MySQL configuration, defines a list of important configuration parameters to export, writes a header with the timestamp, queries MySQL for the current value of each parameter, and writes each parameter and its value to the file. This configuration snapshot is invaluable for troubleshooting or when restoring to a different server as it provides a reference point for the optimal configuration at the time of the backup.<\/p>\n\n\n\n<div class=\"wp-block-urvanov-syntax-highlighter-code-block\"><pre class=\"lang:tsql decode:true \">   \/\/ Export current configuration\n    var configFile = backupDir + '\/mysql_config.txt';\n    var configFileHandle = shell.openFile(configFile, 'w');\n    var configParams = [\n        'innodb_buffer_pool_size',\n        'innodb_buffer_pool_instances',\n        'innodb_log_file_size',\n        'innodb_flush_log_at_trx_commit',\n        'innodb_flush_method',\n        'innodb_io_capacity',\n        'innodb_io_capacity_max',\n        'table_open_cache',\n        'table_definition_cache',\n        'max_connections',\n        'thread_cache_size',\n        'join_buffer_size',\n        'sort_buffer_size',\n        'read_buffer_size',\n        'read_rnd_buffer_size',\n        'key_buffer_size'\n    ];\n    configFileHandle.write(\"MySQL Configuration Backup - \" + timestamp + \"\\n\");\n    configFileHandle.write(\"=\".repeat(50) + \"\\n\\n\");\n    for (var i = 0; i &lt; configParams.length; i++) {\n        var param = configParams[i];\n        var result = session.runSql(\"SELECT @@GLOBAL.\" + param);\n        var value = result.fetchOne()[0];\n        configFileHandle.write(param + \" = \" + value + \"\\n\");\n    }\n    configFileHandle.close();<\/pre><\/div>\n\n\n\n<p>The function then creates a <a href=\"https:\/\/flatfile.com\/blog\/what-is-a-csv-file-guide-to-uses-and-benefits\/\" target=\"_blank\" rel=\"noreferrer noopener\">CSV file<\/a> to store historical performance metrics, writes a header row with column names, queries the monitoring database for all metrics, and writes each row of metrics to the CSV file. Exporting this historical data allows you to analyze performance trends even if the original monitoring database is lost or corrupted.<\/p>\n\n\n\n<div class=\"wp-block-urvanov-syntax-highlighter-code-block\"><pre class=\"lang:tsql decode:true \">   \/\/ Export performance metrics history\n    var metricsFile = backupDir + '\/performance_metrics.csv';\n    var metricsFileHandle = shell.openFile(metricsFile, 'w');\n    metricsFileHandle.write(\"timestamp,buffer_pool_usage,threads_connected,threads_running,slow_queries,table_open_cache_hits,table_open_cache_misses\\n\");\n    var metricsResult = session.runSql(`\n        SELECT \n            timestamp,\n            buffer_pool_usage,\n            threads_connected,\n            threads_running,\n            slow_queries,\n            table_open_cache_hits,\n            table_open_cache_misses\n        FROM \n            performance_monitoring.mysql_metrics\n        ORDER BY \n            timestamp\n    `);\nvar row;\n    while ((row = metricsResult.fetchOne())) {\n        metricsFileHandle.write(\n            row[0] + \",\" + \n            row[1] + \",\" + \n            row[2] + \",\" + \n            row[3] + \",\" + \n            row[4] + \",\" + \n            row[5] + \",\" + \n            row[6] + \"\\n\"\n        );\n    }\n    metricsFileHandle.close();<\/pre><\/div>\n\n\n\n<p>Next, the function creates another CSV file to store configuration change history, exports all configuration changes from the monitoring database, creates yet another CSV file to store database size information, queries the <strong>information_schema<\/strong> to calculate the size of each database, and writes the database names and sizes to the CSV file. The database size information is particularly useful for capacity planning and for determining appropriate buffer pool sizes during restore operations.<\/p>\n\n\n\n<div class=\"wp-block-urvanov-syntax-highlighter-code-block\"><pre class=\"lang:tsql decode:true \">       \/\/ Export configuration changes history\n    var changesFile = backupDir + '\/config_changes.csv';\n    var changesFileHandle = shell.openFile(changesFile, 'w');\n    changesFileHandle.write(\"timestamp,parameter_name,old_value,new_value,change_reason\\n\");\n    var changesResult = session.runSql(`\n        SELECT \n            timestamp,\n            parameter_name,\n            old_value,\n            new_value,\n            change_reason\n        FROM \n            performance_monitoring.config_changes\n        ORDER BY \n            timestamp\n    `);\nwhile ((row = changesResult.fetchOne())) {\n        changesFileHandle.write(\n            row[0] + \",\" + \n            row[1] + \",\" + \n            row[2] + \",\" + \n            row[3] + \",\" + \n            row[4].replace(\",\", \";\") + \"\\n\"\n        );\n    }\n    changesFileHandle.close();\n    \/\/ Export database size information\n    var sizeFile = backupDir + '\/database_sizes.csv';\n    var sizeFileHandle = shell.openFile(sizeFile, 'w');\n    sizeFileHandle.write(\"database_name,size_mb\\n\");\nvar sizeResult = session.runSql(`\n        SELECT \n            table_schema AS database_name,\n            ROUND(SUM(data_length + index_length) \/ 1024 \/ 1024, 2) AS size_mb\n        FROM \n            information_schema.tables\n        GROUP BY \n            table_schema\n        ORDER BY \n            size_mb DESC\n    `);\n    while ((row = sizeResult.fetchOne())) {\n        sizeFileHandle.write(row[0] + \",\" + row[1] + \"\\n\");\n    }\n    sizeFileHandle.close();<\/pre><\/div>\n\n\n\n<p>Finally, the function returns the successful completion of the backup, lists all the files that were created, and returns the backup directory path for potential further processing. This backup includes not just the database data but also sufficient configuration and performance information to help with routine troubleshooting and optimization.<\/p>\n\n\n\n<div class=\"wp-block-urvanov-syntax-highlighter-code-block\"><pre class=\"lang:tsql decode:true \">\/\/ Summary\n    print(\"Backup completed successfully!\");\n    print(\"Database data: \" + backupDir + '\/data');\n    print(\"Configuration: \" + backupDir + '\/mysql_config.txt');\n    print(\"Performance metrics: \" + backupDir + '\/performance_metrics.csv');\n    print(\"Configuration changes: \" + backupDir + '\/config_changes.csv');\n    print(\"Database sizes: \" + backupDir + '\/database_sizes.csv');\n    return backupDir;\n}\nbackupWithConfig();<\/pre><\/div>\n\n\n\n<p>This enriched backup forms the basis for intelligent restores and meaningful performance comparisons.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Recovery with Configuration Analysis<\/h2>\n\n\n\n<p>When restoring a database, it&#8217;s not always appropriate to use the same configuration as before. Let&#8217;s look at a script that restores a database and adjusts its configuration to the current environment.<\/p>\n\n\n\n<p>This function begins by stating the start of the intelligent restore process, opens and reads the configuration file from the backup, parses the file to extract parameter names and values, and displays key configuration values from the backup. This analysis provides a baseline for comparison with the current server&#8217;s capabilities.<\/p>\n\n\n\n<div class=\"wp-block-urvanov-syntax-highlighter-code-block\"><pre class=\"lang:tsql decode:true \">function intelligentRestore(backupDir) {\n    print(\"Starting intelligent restore from: \" + backupDir);\n    \/\/ Read backup configuration file\n    var configFile = backupDir + '\/mysql_config.txt';\n    var configContent = shell.readFile(configFile);\n    var configLines = configContent.split('\\n');\n    var backupConfig = {};\n    for (var i = 0; i &lt; configLines.length; i++) {\n        var line = configLines[i].trim();\n        if (line.indexOf(' = ') &gt; 0) {\n            var parts = line.split(' = ');\n            backupConfig[parts[0]] = parts[1];\n        }\n    }\n    print(\"Backup configuration analyzed:\");\n    print(\"- Buffer pool size: \" + (parseInt(backupConfig.innodb_buffer_pool_size) \/ (1024 * 1024 * 1024)).toFixed(2) + \" GB\");\n    print(\"- Max connections: \" + backupConfig.max_connections);\n    print(\"- Table open cache: \" + backupConfig.table_open_cache);<\/pre><\/div>\n\n\n\n<p>Next, the function reads and parses the database size information from the backup, calculates the total size of all databases, determines the current server&#8217;s capabilities, and displays the total database size and server capabilities. Understanding both the database size and server capabilities is essential for calculating appropriate configuration values.<\/p>\n\n\n\n<div class=\"wp-block-urvanov-syntax-highlighter-code-block\"><pre class=\"lang:tsql decode:true \">   \/\/ Analyze database size\n    var sizeFile = backupDir + '\/database_sizes.csv';\n    var sizeContent = shell.readFile(sizeFile);\n    var sizeLines = sizeContent.split('\\n');\n    var totalDatabaseSizeMB = 0;\n    for (var i = 1; i &lt; sizeLines.length; i++) {\n        var line = sizeLines[i].trim();\n        if (line) {\n            var parts = line.split(',');\n            totalDatabaseSizeMB += parseFloat(parts[1]);\n        }\n    }\n    print(\"Total database size: \" + totalDatabaseSizeMB.toFixed(2) + \" MB (\" + (totalDatabaseSizeMB \/ 1024).toFixed(2) + \" GB)\");\n    \/\/ Simulated server capabilities\n    var currentServerMemoryGB = 16;\n    var currentServerCores = 8;\n    var currentServerDiskType = \"SSD\";\n    print(\"\\nCurrent server capabilities:\");\n    print(\"- Memory: \" + currentServerMemoryGB + \" GB\");\n    print(\"- CPU cores: \" + currentServerCores);\n    print(\"- Disk type: \" + currentServerDiskType);<\/pre><\/div>\n\n\n\n<p>Here, it calculates optimal MySQL configuration settings based on the server&#8217;s capabilities and database size, and recommends a buffer pool size large enough to hold the entire database but capped at 75% of total memory. It also determines the number of buffer pool instances, suitable <a href=\"https:\/\/www.lenovo.com\/gb\/en\/glossary\/what-is-io\/?srsltid=AfmBOooiAPuz6vTVnaSCGqqlvsWEH6qcnFbJguV2ThB3Lek_X1pOOJ4R\" target=\"_blank\" rel=\"noreferrer noopener\">I\/O<\/a> capacity based on disk type, and max connections using both memory and CPU-based heuristics. <\/p>\n\n\n\n<p>Additionally, it sets values for thread cache size (25% of max connections) and table open cache (based on estimated number of tables). These values are printed out for the user to review before proceeding.<\/p>\n\n\n\n<div class=\"wp-block-urvanov-syntax-highlighter-code-block\"><pre class=\"lang:tsql decode:true \">\/\/ Recommend configuration\n    var recommendedConfig = {};\n    var databaseSizeGB = totalDatabaseSizeMB \/ 1024;\n    var recommendedBufferPoolGB = Math.min(\n        currentServerMemoryGB * 0.75,\n        Math.max(databaseSizeGB * 1.2, 1)\n    );\n    recommendedConfig.innodb_buffer_pool_size = Math.round(recommendedBufferPoolGB * 1024 * 1024 * 1024);\n    recommendedConfig.innodb_buffer_pool_instances = Math.min(Math.max(Math.round(recommendedBufferPoolGB), 1), 8);\n    if (currentServerDiskType === \"SSD\") {\n        recommendedConfig.innodb_io_capacity = 2000;\n        recommendedConfig.innodb_io_capacity_max = 4000;\n    } else {\n        recommendedConfig.innodb_io_capacity = 200;\n        recommendedConfig.innodb_io_capacity_max = 400;\n    }\n    var connectionMemoryMB = 10;\n    var maxConnectionsFromMemory = Math.floor((currentServerMemoryGB * 1024 * 0.2) \/ connectionMemoryMB);\n    var maxConnectionsFromCPU = currentServerCores * 25;\n    recommendedConfig.max_connections = Math.min(maxConnectionsFromMemory, maxConnectionsFromCPU);\n    recommendedConfig.thread_cache_size = Math.round(recommendedConfig.max_connections  * 0.25);\n    var tableEstimate = totalDatabaseSizeMB \/ 10;\n    recommendedConfig.table_open_cache = Math.min(Math.max(Math.round(tableEstimate * 1.5), 400), 2000);\n    print(\"\\nRecommended configuration for this server:\");\n    for (var param in recommendedConfig) {\n        print(\"- \" + param + \": \" + recommendedConfig[param]);\n    }<\/pre><\/div>\n\n\n\n<p>Here, the script compares the recommended configuration with the one used in the backup. It prints a side-by-side comparison table showing each parameter, its value in the backup (if available), the newly calculated value, and the reason for any changes. If a parameter is missing in the backup, it&#8217;s marked as such. If the value hasn\u2019t changed, it\u2019s labeled \u201cSame.\u201d Otherwise, the script states that the change was made to align with the current hardware. This step helps users understand how and why the configuration is being tailored for the new environment.<\/p>\n\n\n\n<div class=\"wp-block-urvanov-syntax-highlighter-code-block\"><pre class=\"lang:tsql decode:true \">           \/\/ Comparison table\n    print(\"\\nComparison with backup configuration:\");\n    print(\"Parameter               | Backup Value | Recommended Value | Reason\");\n    print(\"------------------------|--------------|-------------------|------------------\");\n    var compareParams = [\n        { name: \"innodb_buffer_pool_size\", format: function(v) { return (parseInt(v) \/ (1024 * 1024 * 1024)).toFixed(2) + \" GB\"; } },\n        { name: \"innodb_buffer_pool_instances\", format: String },\n        { name: \"innodb_io_capacity\", format: String },\n        { name: \"innodb_io_capacity_max\", format: String },\n        { name: \"max_connections\", format: String },\n        { name: \"thread_cache_size\", format: String },\n        { name: \"table_open_cache\", format: String }\n    ];\n    function getReasonForChange(param, oldVal, newVal) {\n        if (oldVal === \"N\/A\") return \"Missing in backup\";\n        if (String(oldVal) === String(newVal)) return \"Same\";\n        return \"Adjusted for hardware\";\n    }\n    for (var i = 0; i &lt; compareParams.length; i++) {\n        var p = compareParams[i];\n        var name = p.name;\n        var oldVal = backupConfig[name] || \"N\/A\";\n        var newVal = recommendedConfig[name];\n        var reason = getReasonForChange(name, oldVal, newVal);\n        print(name.padEnd(24) + \" | \" +\n              p.format(oldVal).padEnd(12) + \" | \" +\n              p.format(newVal).padEnd(17) + \" | \" + reason);\n    }<\/pre><\/div>\n\n\n\n<p>Before making any changes or restoring data, the script pauses to get user confirmation. If the user does not explicitly respond with \u201cyes,\u201d the restore process is canceled. This safeguard ensures the user has full control and can opt out if the recommended changes don\u2019t look suitable. Once the user approves, the script uses `<strong>util.loadDump()`<\/strong> to restore the database from the backup directory. It ignores the mismatches of the MySQL version and resets the progress counter. This method handles importing all schema and data files stored during the logical backup, effectively recreating the databases on the new server.<\/p>\n\n\n\n<div class=\"wp-block-urvanov-syntax-highlighter-code-block\"><pre class=\"lang:tsql decode:true \">\/\/ Confirmation\n    var answer = shell.prompt(\"\\nDo you want to proceed with the restore using these settings? (yes\/no): \");\n    if (answer.toLowerCase() !== \"yes\") {\n        print(\"Restore cancelled.\");\n        return false;\n\/\/ Restore data\n    print(\"\\nRestoring database data...\");\n    util.loadDump(backupDir + \"\/data\", {\n        ignoreVersion: true,\n        resetProgress: true\n    });<\/pre><\/div>\n\n\n\n<p>Finally, the script applies the recommended configuration settings to the MySQL server using `SET GLOBAL`. Each change is applied one at a time, and the script prints a confirmation message for every parameter successfully set. After this step, the server is both restored and tuned based on current hardware, ensuring optimal performance for the workload.<\/p>\n\n\n\n<div class=\"wp-block-urvanov-syntax-highlighter-code-block\"><pre class=\"lang:tsql decode:true \">   \/\/ Apply config\n    print(\"\\nApplying recommended configuration...\");\n    for (var param in recommendedConfig) {\n        session.runSql(\"SET GLOBAL \" + param + \" = ?\", [recommendedConfig[param]]);\n        print(\"Set \" + param + \" to \" + recommendedConfig[param]);\n    }\n    print(\"\\nRestore and reconfiguration complete.\");\n    return true;\n}<\/pre><\/div>\n\n\n\n<p>With this configuration analysis complete, we\u2019re no longer restoring a snapshot of the past \u2014 we\u2019re provisioning an environment tuned for the present, ready for real testing and confident deployment.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-creating-an-automation-script-for-mysql-shell\">Creating an Automation Script for MySQL Shell<\/h2>\n\n\n\n<p>Now that we&#8217;ve created individual functions for various aspects of MySQL performance tuning, we can combine them all by creating&nbsp;an automation script that ties everything together.<\/p>\n\n\n\n<p>By leveraging automation through MySQL Shell, you\u2019ll spend less time on routine database maintenance and more time doing what matters. Your database will be more reliable, more performant, and easier to manage, even as it scales to handle larger workloads.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Before you begin \u2026.<\/h3>\n\n\n\n<p>This code is a foundation for automating MySQL performance tuning. Before it can be run, you will need to fill in the parts that are currently just placeholders. These placeholders are there to show where you should add your code, because every MySQL setup is unique. What you want to monitor, how you want to tune your database, the backups you want to take, and where you want to store those backups will depend on your specific server environment and your personal preferences.<\/p>\n\n\n\n<p>Regarding the backup folder paths and files, when the script mentions something like &#8216;\/path\/to\/your\/backup&#8217;, this is just a placeholder for the actual folder location on your laptop where you want to save or load backups. You need to change this to a folder path that exists on your system. For example, on <a href=\"https:\/\/www.microsoft.com\/en-gb\/windows?r=1\" target=\"_blank\" rel=\"noreferrer noopener\">Windows<\/a>, it might look like C:\\\\Users\\\\YourName\\\\MySQLBackups\\\\, while on <a href=\"https:\/\/www.apple.com\/uk\/macos\/macos-sequoia\/\" target=\"_blank\" rel=\"noreferrer noopener\">macOS<\/a> or <a href=\"https:\/\/www.linux.org\/\" target=\"_blank\" rel=\"noreferrer noopener\">Linux<\/a>, it might be something like \/home\/username\/mysql_backups\/. If you don\u2019t update this path, the script won\u2019t find the files it needs, and backups won\u2019t save correctly.<\/p>\n\n\n\n<p>To prepare your computer or server for this automation, make sure that the folders you want to use for backups or loading configuration files already exist. The script does not create folders automatically, so this step is essential. Also, if your script needs to read files like configuration files or backup files, those files must be in the exact locations you specify in the script. Make sure to update the script to point precisely to those folder paths.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-the-key-components-of-the-automation-system\">The Key Components of the Automation System<\/h3>\n\n\n\n<p>The key components of our automation system that I\u2019ve covered in this series of articles are:<\/p>\n\n\n<div class=\"block-core-list\">\n<ol start=\"1\" class=\"wp-block-list\">\n<li>`<strong>setupMonitoringDatabase()`:<\/strong> Creates the necessary database and tables for storing metrics and configuration changes<br><\/li>\n\n\n\n<li>`<strong>collectMySQLMetrics()`:<\/strong> Gathers performance metrics from MySQL and stores them in the database<br><\/li>\n\n\n\n<li>`<strong>setupPrometheusEndpoint()`:<\/strong> Exposes MySQL metrics in a format that <a href=\"https:\/\/prometheus.io\/\" target=\"_blank\" rel=\"noreferrer noopener\">Prometheus<\/a> can scrape<br><\/li>\n\n\n\n<li>`<strong>adjustBufferPoolSize()<\/strong>`: Analyzes buffer pool usage and adjusts its size accordingly<br><\/li>\n\n\n\n<li><strong>`analyzePerformanceTrends()<\/strong>`: Generates reports of performance trends over time<br><\/li>\n\n\n\n<li><strong>`backupWithConfig()<\/strong>`: Creates backups that include configurations<br><\/li>\n\n\n\n<li>`<strong>intelligentRestore()<\/strong>`: Restores a database and adapts its configuration to the current environment<br><\/li>\n\n\n\n<li>`<strong>automatePerformanceTuning()<\/strong>`: Ties everything together into a comprehensive automation solution<\/li>\n<\/ol>\n<\/div>\n\n\n<h3 class=\"wp-block-heading\" id=\"h-building-the-automation-script\">Building the Automation Script <\/h3>\n\n\n\n<p>This function begins by setting default intervals for monitoring (5 minutes), tuning (1 hour), reporting (1 day), and backup (1 week), but allows these to be customized through a configuration object. It then ensures the monitoring database exists by calling our `<strong>setupMonitoringDatabase()`<\/strong> function.<\/p>\n\n\n\n<div class=\"wp-block-urvanov-syntax-highlighter-code-block\"><pre class=\"lang:tsql decode:true \">function automatePerformanceTuning(config) {\n    \/\/ Default configuration\n    config = config || {};\n    config.monitoringInterval = config.monitoringInterval || 300;    \/\/ 5 minutes\n    config.tuningInterval = config.tuningInterval || 3600;           \/\/ 1 hour\n    config.reportingInterval = config.reportingInterval || 86400;    \/\/ 1 day\n    config.backupInterval = config.backupInterval || 604800;         \/\/ 1 week\n    print(\"Starting comprehensive MySQL performance automation...\");\n    print(\"Monitoring interval: \" + config.monitoringInterval + \" seconds\");\n    print(\"Tuning interval: \" + config.tuningInterval + \" seconds\");\n    print(\"Reporting interval: \" + config.reportingInterval + \" seconds\");\n    print(\"Backup interval: \" + config.backupInterval + \" seconds\");\n    \/\/ Ensure monitoring database exists\n    print(\"\\nSetting up monitoring database...\");\n    if (!setupMonitoringDatabase()) {\n        print(\"Failed to set up monitoring database. Exiting automation.\");\n        return;\n    }<\/pre><\/div>\n\n\n\n<p>The function initializes timers to track when each task was last performed, and then enters a continuous loop. It checks if it is time to collect metrics based on the monitoring interval and calls our `<strong>collectMySQLMetrics()`<\/strong> function if it is.<\/p>\n\n\n\n<div class=\"wp-block-urvanov-syntax-highlighter-code-block\"><pre class=\"lang:tsql decode:true \">\/\/ Initialize timers\n    var lastMonitoringTime = 0;\n    var lastTuningTime = 0;\n    var lastReportingTime = 0;\n    var lastBackupTime = 0;\n    \/\/ Main automation loop\n    print(\"\\nAutomation running. Press Ctrl+C to stop.\");\n    while (true) {\n        var currentTime = Math.floor(Date.now() \/ 1000);\n        \/\/ Collect metrics\n        if (currentTime - lastMonitoringTime &gt;= config.monitoringInterval) {\n            print(\"\\n\" + new Date().toISOString() + \" - Collecting performance metrics...\");\n            collectMySQLMetrics();\n            lastMonitoringTime = currentTime;\n        }<\/pre><\/div>\n\n\n\n<p>The function then checks if it&#8217;s time to perform tuning based on the tuning interval. If it is, it calls our `<strong>adjustBufferPoolSize()<\/strong>` and `<strong>adjustTableOpenCache()<\/strong>` functions. If any configuration changes are made, it immediately collects new metrics to measure the impact of those changes.<\/p>\n\n\n\n<div class=\"wp-block-urvanov-syntax-highlighter-code-block\"><pre class=\"lang:tsql decode:true \">       \/\/ Perform tuning\n        if (currentTime - lastTuningTime &gt;= config.tuningInterval) {\n            print(\"\\n\" + new Date().toISOString() + \" - Performing performance tuning...\");\n            var bufferPoolChanged = adjustBufferPoolSize();\n            var tableCacheChanged = adjustTableOpenCache();\n            if (bufferPoolChanged || tableCacheChanged) {\n                print(\"Configuration changes were made. Collecting new metrics...\");\n                collectMySQLMetrics();\n            }\n            lastTuningTime = currentTime;\n        }<\/pre><\/div>\n\n\n\n<p>Finally, the function does two checks at each interval and the script performs either of these key actions:<\/p>\n\n\n<div class=\"block-core-list\">\n<ul class=\"wp-block-list\">\n<li>If it&#8217;s time to generate a report based on the reporting interval, it calls our `<strong>analyzePerformanceTrends()`<\/strong> function.<br><\/li>\n\n\n\n<li>If it&#8217;s time to perform a backup based on the backup interval, it calls our `<strong>backupWithConfig()`<\/strong> function.<\/li>\n<\/ul>\n<\/div>\n\n\n<p>The function then sleeps briefly to avoid high CPU usage. This allows the automation script to provide a solid foundation for keeping your MySQL server optimized around the clock.<\/p>\n\n\n\n<div class=\"wp-block-urvanov-syntax-highlighter-code-block\"><pre class=\"lang:tsql decode:true \">\/\/ Generate reports\n        if (currentTime - lastReportingTime &gt;= config.reportingInterval) {\n            print(\"\\n\" + new Date().toISOString() + \" - Generating performance report...\");\n            analyzePerformanceTrends(7); \/\/ Past 7 days\n            lastReportingTime = currentTime;\n        }\n        \/\/ Perform backup\n        if (currentTime - lastBackupTime &gt;= config.backupInterval) {\n            print(\"\\n\" + new Date().toISOString() + \" - Performing backup with configuration tracking...\");\n            backupWithConfig();\n            lastBackupTime = currentTime;\n        }\n        \/\/ Sleep for 1 second to avoid high CPU usage\n        shell.wait(1);\n    }\n}<\/pre><\/div>\n\n\n\n<p>You can start the automation with default settings:<\/p>\n\n\n\n<div class=\"wp-block-urvanov-syntax-highlighter-code-block\"><pre class=\"lang:tsql decode:true \">\/\/ Start automation with default settings\nautomatePerformanceTuning();\nOr customize the Intervals:\n\/\/ Start automation with custom intervals\nautomatePerformanceTuning({\n    monitoringInterval: 60,    \/\/ Collect metrics every minute\n    tuningInterval: 1800,      \/\/ Tune every 30 minutes\n    reportingInterval: 43200,  \/\/ Generate reports every 12 hours\n    backupInterval: 259200     \/\/ Backup every 3 days\n});<\/pre><\/div>\n\n\n\n<p>When executed, the automation script will produce continuous output as it collects metrics, performs tuning, generates reports, and creates backups.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-recommended-articles-on-mysql-performance\">Recommended Articles on MySQL Performance<\/h2>\n\n\n\n<p>If you&#8217;re interested in learning more about MySQL memory management, performance tuning, and best practices for optimizing the InnoDB buffer pool, the following resources are great places to dive deeper:<\/p>\n\n\n<div class=\"block-core-list\">\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/www.percona.com\/blog\/tuning-mysql-innodb-flushing-for-a-write-intensive-workload\/\" target=\"_blank\" rel=\"noreferrer noopener\">InnoDB Buffer Pool Tuning for Write-Heavy Workloads<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/docs.akamas.io\/akamas-docs\/knowledge-base\/optimizing-a-mysql-server-database-running-oltpbench\" target=\"_blank\" rel=\"noreferrer noopener\">Optimizing MySQL Configuration for OLTP Workloads<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/shrihariharidas73.medium.com\/unlocking-database-insights-monitoring-mysql-with-prometheus-and-grafana-ddd2c4f01929\" target=\"_blank\" rel=\"noreferrer noopener\">Monitoring MySQL with Prometheus and Grafana<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/dev.mysql.com\/doc\/refman\/8.0\/en\/optimize-overview.html\" target=\"_blank\" rel=\"noreferrer noopener\">MySQL Query Performance Tuning<\/a><\/li>\n<\/ul>\n<\/div>\n\n\n<h2 class=\"wp-block-heading\" id=\"h-conclusion-and-next-steps\">Conclusion and Next Steps<\/h2>\n\n\n\n<p>I hope that the techniques that I\u2019ve explained in <a href=\"https:\/\/www.red-gate.com\/simple-talk\/collections\/chisom-kanus-series-mysql-shell-and-basic-configuration-management\/\" target=\"_blank\" rel=\"noreferrer noopener\">this series<\/a> serve you well as you build and maintain high-performance database environments that can scale with your growing needs.<\/p>\n\n\n\n<p>The series has taken you from the <a href=\"https:\/\/www.red-gate.com\/simple-talk\/databases\/mysql\/part-i-introduction-to-mysql-shell-and-basic-configuration-management\/\" target=\"_blank\" rel=\"noreferrer noopener\">basics of MySQL Shell<\/a> all the way to automation techniques. The foundation we&#8217;ve built is extensible and adaptable: you can add new metrics, implement additional optimization algorithms, integrate with other tools in your infrastructure, or customize the decision-making logic to match your specific requirements.<\/p>\n\n\n\n<p>In this final article, we&#8217;ve created a complete automation system for monitoring, analyzing, and optimizing MySQL performance using MySQL Shell. This system collects performance metrics at regular intervals, stores them in a dedicated database, analyzes them to identify optimization opportunities, automatically adjusts MySQL configuration based on actual usage patterns, and provides tools for analyzing performance trends.<br><br>By implementing this automation system, you can ensure that your MySQL server remains optimized even as conditions change. The system continuously monitors performance, makes data-driven decisions about configuration changes, and provides valuable insights into performance trends over time. By writing scripts like the ones we&#8217;ve demonstrated, you can automate nearly every aspect of MySQL performance tuning and management. This automation not only saves time but also ensures that your MySQL servers remain optimized even as conditions change.<\/p>\n\n\n\n<p>Remember that this is just a starting point. You can extend the system to monitor and optimize additional MySQL parameters, integrate with external monitoring tools like <a href=\"https:\/\/prometheus.io\/\" target=\"_blank\" rel=\"noreferrer noopener\">Prometheus<\/a> and <a href=\"https:\/\/grafana.com\/\" target=\"_blank\" rel=\"noreferrer noopener\">Grafana<\/a>, or add more analysis and decision-making logic. Of course, automation isn\u2019t something you set once and forget. As your environment evolves, your scripts will need to evolve too. The examples we\u2019ve covered are meant to be a starting point &#8211; a solid foundation you can build on to create a performance tuning system made to your specific needs.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Learn how to automate MySQL performance tuning using MySQL Shell. Discover intelligent backup and recovery strategies, configuration tracking, and optimization scripts.&hellip;<\/p>\n","protected":false},"author":342096,"featured_media":107554,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[143523,145792],"tags":[159266,5854],"coauthors":[159002],"class_list":["post-107551","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-databases","category-mysql","tag-chisomkanu_mysqlshell","tag-mysql"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/107551","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/users\/342096"}],"replies":[{"embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/comments?post=107551"}],"version-history":[{"count":5,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/107551\/revisions"}],"predecessor-version":[{"id":107561,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/107551\/revisions\/107561"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/media\/107554"}],"wp:attachment":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/media?parent=107551"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/categories?post=107551"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/tags?post=107551"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/coauthors?post=107551"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}